大规模利用组织数据
Leveraging Organizational Data at Scale
版权所有 © 2020 亚当·贝尔马尔。版权所有。
Copyright © 2020 Adam Bellemare. All rights reserved.
美国印刷。
Printed in the United States of America.
由O'Reilly Media, Inc.出版,地址:1005 Gravenstein Highway North, Sebastopol, CA 95472。
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
购买 O'Reilly 书籍可用于教育、商业或促销用途。大多数图书也提供在线版本 ( http://oreilly.com )。欲了解更多信息,请联系我们的企业/机构销售部门:800-998-9938或corporate@oreilly.com。
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.
有关发布详细信息,请参阅http://oreilly.com/catalog/errata.csp?isbn=9781492057895。
See http://oreilly.com/catalog/errata.csp?isbn=9781492057895 for release details.
O'Reilly 徽标是 O'Reilly Media, Inc. 的注册商标。构建事件驱动的微服务、封面图片和相关商业外观是 O'Reilly Media, Inc. 的商标。
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Building Event-Driven Microservices, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
本文所表达的观点仅代表作者个人观点,并不代表出版商的观点。虽然出版商和作者已尽力确保本作品中包含的信息和说明准确无误,但出版商和作者不承担任何错误或遗漏的责任,包括但不限于因使用或使用而造成的损害的责任对这项工作的依赖。使用本作品中包含的信息和说明的风险由您自行承担。如果本作品包含或描述的任何代码示例或其他技术受开源许可证或他人知识产权的约束,则您有责任确保您对其的使用符合此类许可证和/或权利。
The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
978-1-492-05789-5
978-1-492-05789-5
[大规模集成电路]
[LSI]
我写这本书是为了当我开始进入事件驱动的微服务世界时我希望能拥有这本书。这本书是我自己的个人经历、与他人的讨论以及与事件驱动的微服务世界的某一部分相关的无数博客、书籍、帖子、演讲、会议和文档的结晶。我发现我读到的许多著作都只是顺便提到了事件驱动架构,或者深度不够。有些只涵盖了架构的特定方面,虽然有帮助,但只提供了难题的一小部分。其他工作被证明是简化和不屑一顾的,断言事件驱动系统实际上只对一个系统有用,可以将异步消息直接发送到另一个系统,作为同步请求响应系统的替代品。
I wrote this book to be the book that I wish I’d had when I started out on my journey into the world of event-driven microservices. This book is a culmination of my own personal experiences, discussions with others, and the countless blogs, books, posts, talks, conferences, and documentation related to one part or another of the event-driven microservice world. I found that many of the works I read mentioned event-driven architectures either only in passing or with insufficient depth. Some covered only a specific aspect of the architecture and, while helpful, provided only a small piece of the puzzle. Other works proved to be reductive and dismissive, asserting that event-driven systems are really only useful for one system to send an asynchronous message directly to another as a replacement for synchronous request-response systems. As this book details, there is far more to event-driven architectures than this.
我们使用的工具对我们的发明产生了重大影响。事件驱动的微服务架构是由一系列最近才变得容易使用的技术实现的。分布式、容错、高容量和高速事件代理是本书架构和设计模式的基础。这些技术解决方案基于大数据与近实时事件处理需求的融合。容器化的便利性和计算资源的需求促进了微服务的发展,从而简化了数十万个微服务的托管、扩展和管理。
The tools that we use shape and influence our inventions significantly. Event-driven microservice architectures are made possible by a whole host of technologies that have only recently become readily accessible. Distributed, fault-tolerant, high-capacity, and high-speed event brokers underpin the architectures and design patterns in this book. These technological solutions are based on the convergence of big data with the need for near-real-time event processing. Microservices are facilitated by the ease of containerization and the requisitioning of compute resources, allowing for simplified hosting, scaling, and management of hundreds of thousands of microservices.
支持事件驱动的微服务的技术对我们思考和解决问题的方式以及我们的业务和组织的结构具有重大影响。事件驱动的微服务改变了业务的运作方式、问题的解决方式以及团队、人员和业务部门的沟通方式。这些工具为您提供了一种真正新的做事方式,直到最近才成为可能。
The technologies that support event-driven microservices have a significant impact on how we think about and solve problems, as well as on how our businesses and organizations are structured. Event-driven microservices change how a business works, how problems can be solved, and how teams, people, and business units communicate. These tools give you a truly new way of doing things that has not been possible until only recently.
本书使用以下印刷约定:
The following typographical conventions are used in this book:
表示新术语、URL、电子邮件地址、文件名和文件扩展名。
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant widthConstant width用于程序列表,以及在段落中引用程序元素,例如变量或函数名称、数据库、数据类型、环境变量、语句和关键字。
Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Constant width boldConstant width bold显示应由用户逐字键入的命令或其他文本。
Shows commands or other text that should be typed literally by the user.
Constant width italicConstant width italic显示应替换为用户提供的值或上下文确定的值的文本。
Shows text that should be replaced with user-supplied values or by values determined by context.
该元素表示提示或建议。
This element signifies a tip or suggestion.
该元素表示一般注释。
This element signifies a general note.
该元素表示警告或警告。
This element indicates a warning or caution.
40 多年来,O'Reilly Media一直提供技术和业务培训、知识和见解来帮助公司取得成功。
For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed.
我们独特的专家和创新者网络通过书籍、文章和我们的在线学习平台分享他们的知识和专业知识。O'Reilly 的在线学习平台让您可以按需访问实时培训课程、深入学习路径、交互式编码环境以及来自 O'Reilly 和 200 多家其他出版商的大量文本和视频。欲了解更多信息,请访问http://oreilly.com。
Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.
请向出版商提出有关本书的意见和问题:
Please address comments and questions concerning this book to the publisher:
我们有本书的网页,其中列出了勘误表、示例和任何其他信息。您可以通过https://oreil.ly/building-event-driven-microservices访问此页面。
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/building-event-driven-microservices.
发送电子邮件至bookquestions@oreilly.com发表评论或询问有关本书的技术问题。
Email bookquestions@oreilly.com to comment or ask technical questions about this book.
有关我们的书籍和课程的新闻和信息,请访问http://oreilly.com。
For news and information about our books and courses, visit http://oreilly.com.
在 Facebook 上找到我们: http: //facebook.com/oreilly
Find us on Facebook: http://facebook.com/oreilly
在 Twitter 上关注我们: http: //twitter.com/oreillymedia
Follow us on Twitter: http://twitter.com/oreillymedia
在 YouTube 上观看我们的视频: http: //youtube.com/oreillymedia
Watch us on YouTube: http://youtube.com/oreillymedia
我想对 Confluence 的员工表示敬意和感谢,他们发明了 Apache Kafka,也是最早在事件驱动架构方面“理解”的人之一。我很幸运能得到他们的成员之一 Ben Stopford(首席技术官办公室首席技术专家)提供的充足且有价值的反馈。PHEMI Systems 的首席技术官 Scott Morrison 也为我提供了宝贵的见解、反馈和建议。我向斯科特和本致以谢意,感谢他们帮助使这本书成为今天的样子。作为主要校对者和技术专家,他们帮助我完善想法,要求我提高内容质量,阻止我传播错误的信息,并帮助我讲述事件驱动架构的故事。
I’d like to express my respect and gratitude for the people at Confluent, who, along with inventing Apache Kafka, are some of the first people who particularly “get it” when it comes to event-driven architectures. I have been fortunate enough to have one of their members, Ben Stopford (lead technologist, Office of the CTO), provide ample and valuable feedback. Scott Morrison, CTO of PHEMI Systems, has also provided me with valuable insights, feedback, and recommendations. I offer my thanks and gratitude to both Scott and Ben for helping make this book what it is today. As primary proofreaders and technical experts, they have helped me refine ideas, challenged me to improve the content quality, prevented me from promoting incorrect information, and helped me tell the story of event-driven architectures.
我还要感谢我的朋友贾斯汀·托卡丘克 (Justin Tokarchuk)、加里·格雷厄姆 (Gary Graham) 和尼克·格林 (Nick Green),他们校对和编辑了我的一些草稿。他们与斯科特和本一起帮助我找出叙述中最重要的弱点,提出改进方法,并提供与材料相关的见解和个人经验。
I would also like to extend my thanks to my friends Justin Tokarchuk, Gary Graham, and Nick Green, who proofread and edited a number of my drafts. Along with Scott and Ben, they helped me to identify the most significant weak points in my narrative, suggested ways to improve them, and provided their insights and personal experience in relation to the material.
我还要感谢 O'Reilly 的工作人员以多种方式帮助我。在这段经历中,我与许多优秀的人一起工作过,但我特别要感谢我的编辑 Corbin Collins,他支持我度过了一些困难时期,并帮助我走上正轨。在这一努力中,他是一位出色的合作者,我很感谢他为支持我所做的努力。
My thanks also goes out to the folks at O’Reilly for helping me in innumerable ways. I have worked with a number of excellent people during this experience, but in particular I would like to thank my editor, Corbin Collins, for supporting me through some difficult times and helping keep me on track. He has been a great collaborator during this endeavor, and I appreciate the efforts he has put into supporting me.
我的文案编辑雷切尔·莫纳汉(Rachel Monaghan)让我想起了我的高中时代,当时我的论文会被退回,并带有红色突出显示。我非常感谢她敏锐的眼光和英语知识——她帮助使这本书更容易阅读和理解。谢谢你,雷切尔。
Rachel Monaghan, my copyeditor, reminded me of my high school days, when my essays would be returned colored with red highlights. I am extremely grateful for her sharp eye and knowledge of the English language—she helped make this book far easier to read and understand. Thank you, Rachel.
Christopher Faucher 对我非常耐心,为我提供了极好的反馈,并允许我在最后一刻毫不眨眼地对本书进行了一些重要的修改。谢谢你,克里斯。
Christopher Faucher has been very patient with me, providing me with excellent feedback and allowing me to make a number of nontrivial, last-minute changes to the book without blinking an eye. Thank you, Chris.
内容策略副总裁 Mike Loukides 是我在 O'Reilly 的首批联系人之一。当我向他提出异常冗长的提案时,他耐心地与我一起重新调整它的重点,并将其完善为今天摆在你们面前的这本书的基础。我很感激他花时间与我合作并最终推进这项工作。我已尽力听取他的警告,避免写出一本长度与字典相媲美的巨著。
Mike Loukides, VP of Content Strategy, was one of my first contacts at O’Reilly. When I approached him with my exceptionally verbose and lengthy proposal, he patiently worked with me to refocus it and refine it into the basis of the book before you today. I am grateful that he took the time to work with me and eventually move forward with this work. I have tried my best to heed his caution to avoid producing a tome that rivals the dictionary in length.
感谢我的父母,感谢你们给予我对文字的欣赏。我感谢他们的爱和支持。我的父亲把我介绍给了马歇尔·麦克卢汉,虽然我基本上没有读过他的大部分作品,但我对他对媒介如何影响信息的评价感到非常欣赏。这改变了我看待和评估系统架构的方式。
To my mother and father, I thank you for giving me appreciation for the written word. I am grateful for their love and support. My father introduced me to Marshall McLuhan, and though I have largely failed to read most of his works, I have gained an immense appreciation for his evaluation on how the medium affects the message. This has transformed the way that I view and evaluate system architectures.
最后,感谢所有以某种或大或小的方式为支持我和这项工作做出贡献的人。有很多人以自己的方式做出了贡献——通过对话、博客文章、演示文稿、开源代码、轶事、个人经历、故事和即兴言论。谢谢你们每一个人。
Finally, thanks to everyone else who contributed in some way large or small to supporting me and this work. There are so many people who have contributed in their own way—through conversations, blog posts, presentations, open source code, anecdotes, personal experiences, stories, and impromptu rants. Thank you, each and every one of you.
写这本书既令人高兴又令人沮丧。有很多次我咒骂自己开始这样做,但值得庆幸的是,有很多次我很高兴我这么做了。亲爱的读者,我希望这本书能够以某种方式帮助您学习和成长。
It has been both a pleasure and a frustration to work on this book. There have been many times where I cursed myself for starting it, but thankfully there were many more times that I was glad I did. I hope that this book helps you, dear reader, in some way to learn and grow.
媒介就是消息。
马歇尔·麦克卢汉
The medium is the message.
Marshall McLuhan
麦克卢汉认为,影响人类并给社会带来根本性变化的不是媒体的内容,而是人们对媒体的参与。由于我们的集体参与,报纸、广播、电视、互联网、即时通讯和社交媒体都改变了人类互动和社会 结构。
McLuhan argues that it is not the content of media, but rather engagement with its medium, that impacts humankind and introduces fundamental changes to society. Newspapers, radio, television, the internet, instant messaging, and social media have all changed human interaction and social structures thanks to our collective engagement.
计算机系统架构也是如此。您只需查看计算发明的历史,就可以了解网络通信、关系数据库、大数据开发和云计算如何显着改变架构的构建方式和工作的执行方式。这些发明中的每一项不仅改变了技术在各种软件项目中使用的方式,而且改变了组织、团队和人们彼此沟通的方式。从集中式大型机到分布式移动应用程序,每种新媒介都从根本上改变了人们与计算的关系。
The same is true with computer system architectures. You need only look at the history of computing inventions to see how network communications, relational databases, big-data developments, and cloud computing have significantly altered how architectures are built and how work is performed. Each of these inventions changed not only the way that technology was used within various software projects, but also the way that organizations, teams, and people communicated with one another. From centralized mainframes to distributed mobile applications, each new medium has fundamentally changed people’s relationship with computing.
现代技术已经从根本上改变了异步生产和消费事件的媒介。这些事件现在可以无限期地、大规模地持续存在,并且可以根据需要被任何服务使用多次。计算资源可以轻松获取和按需释放,从而轻松创建和管理微服务。微服务可以根据自己的需求存储和管理数据,其规模以前仅限于基于批处理的大数据解决方案。这些对简单的事件驱动媒介的改进具有深远的影响,不仅改变了计算机架构,而且彻底重塑了团队、人员和组织创建系统和业务的方式。
The medium of the asynchronously produced and consumed event has been fundamentally shifted by modern technology. These events can now be persisted indefinitely, at extremely large scale, and be consumed by any service as many times as necessary. Compute resources can be easily acquired and released on-demand, enabling the easy creation and management of microservices. Microservices can store and manage their data according to their own needs, and do so at a scale that was previously limited to batch-based big-data solutions. These improvements to the humble and simple event-driven medium have far-reaching impacts that not only change computer architectures, but also completely reshape how teams, people, and organizations create systems and businesses.
微服务和微服务风格的架构已经以许多不同的形式、不同的名称存在了很多年。面向服务的架构 (SOA) 通常由多个同步直接相互通信的微服务组成。消息传递架构使用可消费事件来相互异步通信。基于事件的通信当然不是什么新鲜事,但大规模、实时处理大数据集的需求是新的,并且需要改变旧的架构风格。
Microservices and microservice-style architectures have existed for many years, in many different forms, under many different names. Service-oriented architectures (SOAs) are often composed of multiple microservices synchronously communicating directly with one another. Message-passing architectures use consumable events to asynchronously communicate with one another. Event-based communication is certainly not new, but the need for handling big data sets, at scale and in real time, is new and necessitates a change from the old architectural styles.
在现代事件驱动的微服务架构中,系统通过发布和消费事件进行通信。这些事件不会像消息传递系统中那样在使用时被销毁,而是仍然可供其他消费者根据需要读取。这是一个重要的区别,因为它允许本书中涵盖的真正强大的模式。
In a modern event-driven microservices architecture, systems communicate by issuing and consuming events. These events are not destroyed upon consumption as in message-passing systems, but instead remain readily available for other consumers to read as they require. This is an important distinction, as it allows for the truly powerful patterns covered in this book.
这些服务本身规模较小,并且是专门构建的,旨在帮助实现组织的必要业务目标。“小”的典型定义是写的时间不超过两周。根据另一个定义,服务应该能够(概念上)适合人们自己的头脑。这些服务使用来自输入事件流的事件;应用其特定的业务逻辑;并且可以发出自己的输出事件、提供请求响应访问的数据、与第三方 API 通信或执行其他所需的操作。正如本书将详细介绍的,这些服务可以是有状态的或无状态的、复杂的或简单的;它们可以作为长期运行的独立应用程序实现,或者使用函数即服务作为函数执行。
The services themselves are small and purpose-built, created to help fulfill the necessary business goals of the organization. A typical definition of “small” is something that takes no more than two weeks to write. By another definition, the service should be able to (conceptually) fit within one’s own head. These services consume events from input event streams; apply their specific business logic; and may emit their own output events, provide data for request-response access, communicate with a third-party API, or perform other required actions. As this book will detail, these services can be stateful or stateless, complex or simple; and they might be implemented as long-running, standalone applications or executed as a function using Functions-as-a-Service.
事件流和微服务的这种组合形成了整个业务组织的互连活动图。传统的计算机架构由单体和单体间通信组成,具有类似的图结构。这两个图均如图 1-1所示。
This combination of event streams and microservices forms an interconnected graph of activity across a business organization. Traditional computer architectures, composed of monoliths and intermonolith communications, have a similar graph structure. Both of these graphs are shown in Figure 1-1.
确定如何使该图结构有效运行涉及查看两个主要组件:节点和连接。本章将依次研究每一个。
Identifying how to make this graph structure operate efficiently involves looking at the two major components: the nodes and connections. This chapter will examine each in turn.
领域驱动设计由 Eric Evans 在他的同名书中提出,介绍了构建事件驱动的微服务的一些必要概念。鉴于有大量文章、书籍和本月博客可供讨论这个主题,我将保持本节的简短。
Domain-driven design, as coined by Eric Evans in his book of the same title, introduces some of the necessary concepts for building event-driven microservices. Given the wealth of articles, books, and blogs-of-the-month readily available to talk about this subject, I will keep this section brief.
以下概念支撑领域驱动设计:
The following concepts underpin domain-driven design:
企业占据并提供解决方案的问题空间。这涵盖了业务必须应对的所有内容,包括规则、流程、想法、特定于业务的术语以及与其问题空间相关的任何内容,无论业务本身是否与之相关。无论业务是否存在,域都存在。
The problem space that a business occupies and provides solutions to. This encompasses everything that the business must contend with, including rules, processes, ideas, business-specific terminology, and anything related to its problem space, regardless of whether or not the business concerns itself with it. The domain exists regardless of the existence of the business.
主域的组成部分。每个子域都专注于特定的职责子集,通常反映一些业务的组织结构(例如仓库、销售和工程)。子域本身可以被视为域。子域与域本身一样,属于问题空间。
A component of the main domain. Each subdomain focuses on a specific subset of responsibilities and typically reflects some of the business’s organizational structure (such as Warehouse, Sales, and Engineering). A subdomain can be seen as a domain in its own right. Subdomains, like the domain itself, belong to the problem space.
对商业目的有用的实际域的抽象。对业务最重要的领域部分和属性用于生成模型。企业的主要领域模型可以通过企业向客户提供的产品、客户与产品交互的界面以及企业实现其既定目标的各种其他流程和功能来识别。随着领域的变化和业务优先级的变化,模型通常需要进行完善。领域模型是解决方案空间的一部分,因为它是业务用来解决问题的构造。
An abstraction of the actual domain useful for business purposes. The pieces and properties of the domain that are most important to the business are used to generate the model. The main domain model of an business is discernible through the products the business provides its customers, the interfaces by which customers interact with the products, and the various other processes and functions by which the business fulfills its stated goals. Models often need to be refined as the domain changes and as business priorities shift. A domain model is part of the solution space, as it is a construct the business uses to solve problems.
逻辑边界,包括与子域相关的输入、输出、事件、需求、流程和数据模型。虽然理想情况下,有界上下文和子域将完全一致,但遗留系统、技术债务和第三方集成通常会产生例外。有界上下文也是解决方案空间的一种属性,对微服务之间的交互方式有重大影响。
The logical boundaries, including the inputs, outputs, events, requirements, processes, and data models, relevant to the subdomain. While ideally a bounded context and a subdomain will be in complete alignment, legacy systems, technical debt, and third-party integrations often create exceptions. Bounded contexts are also a property of the solution space and have a significant impact on how microservices interact with one another.
有界上下文应该具有高度内聚性。上下文的内部运作应该是密集且密切相关的,绝大多数的沟通发生在内部而不是跨界。具有高度内聚的职责可以缩小设计范围并简化实现。
Bounded contexts should be highly cohesive. The internal operations of the context should be intensive and closely related, with the vast majority of communication occurring internally rather than cross-boundary. Having highly cohesive responsibilities allows for reduced design scope and simpler implementations.
有界上下文之间的连接应该是松散耦合的,因为在一个有界上下文中所做的更改应该最小化或消除对相邻上下文的影响。松耦合可以确保一个上下文中的需求更改不会将大量依赖更改传播到相邻上下文。
Connections between bounded contexts should be loosely coupled, as changes made within one bounded context should minimize or eliminate the impact on neighboring contexts. A loose coupling can ensure that requirement changes in one context do not propagate a surge of dependent changes to neighboring contexts.
每个组织在自身和外部世界之间形成一个单一的域。组织内工作的每个人都在努力支持其领域的需求。
Every organization forms a single domain between itself and the outside world. Everyone working within the organization is operating to support the needs of its domain.
该域被分解为子域——对于以技术为中心的公司来说,可能是工程部门、销售部门和客户支持部门。每个子域都有自己的要求和职责,并且本身可以细分。重复此划分过程,直到子域模型变得细粒度且可操作,并且可以由实施团队转换为小型且独立的服务。有界上下文是围绕这些子域建立的,并构成创建微服务的基础。
This domain is broken down into subdomains—perhaps, for a technology-centric company, an Engineering department, a Sales department, and a Customer Support department. Each subdomain has its own requirements and duties and may itself be subdivided. This division process repeats until the subdomain models are granular and actionable and can be translated into small and independent services by the implementing teams. Bounded contexts are established around these subdomains and form the basis for the creation of microservices.
产品的业务需求在其生命周期内发生变化是很常见的,可能是由于组织变化或新功能请求。相比之下,很少有公司需要更改任何给定产品的底层实现而不伴随业务需求更改。这就是为什么有界上下文应该围绕业务需求而不是技术需求构建。
It is common for the business requirements of a product to change during its lifetime, perhaps due to organizational changes or new feature requests. In contrast, it’s rare for a company to need to change the underlying implementation of any given product without accompanying business requirement changes. This is why bounded contexts should be built around business requirements and not technological requirements.
根据业务需求调整有界上下文,使团队能够以松散耦合和高度内聚的方式对微服务实现进行更改。它为团队提供了针对特定业务需求设计和实施解决方案的自主权,这大大减少了团队间的依赖性,并使每个团队能够严格关注自己的需求。
Aligning bounded contexts on business requirements allows teams to make changes to microservice implementations in a loosely coupled and highly cohesive way. It provides a team with the autonomy to design and implement a solution for the specific business needs, which greatly reduces interteam dependencies and enables each team to focus strictly on its own requirements.
相反,根据技术要求调整微服务是有问题的。这种模式经常出现在设计不当的同步点对点微服务和传统的整体式计算系统中,其中团队拥有应用程序的特定技术层。技术协调的主要问题是,它在多个有界上下文中分配了履行业务功能的责任,这可能涉及具有不同时间表和职责的多个团队。由于没有团队单独负责实施解决方案,因此每个服务都会跨团队和 API 边界与另一个服务耦合,从而使更改变得困难且昂贵。看似无害的改变,一个错误,或者一个失败的服务可能会对使用该技术系统的所有服务的业务服务能力产生严重的连锁反应。技术对齐很少用于事件驱动的微服务 (EDM) 架构中,应尽可能完全避免。消除跨领域的技术和团队依赖性将降低系统对变化的敏感性。
Conversely, aligning microservices on technical requirements is problematic. This pattern is often seen in improperly designed synchronous point-to-point microservices and in traditional monolith-style computing systems where teams own specific technical layers of the application. The main issue with technological alignment is that it distributes the responsibility of fulfilling the business function across multiple bounded contexts, which may involve multiple teams with differing schedules and duties. Because no team is solely responsible for implementing a solution, each service becomes coupled to another across both team and API boundaries, making changes difficult and expensive. A seemingly innocent change, a bug, or a failed service can have serious ripple effects to the business-serving capabilities of all services that use the technical system. Technical alignment is seldomly used in event-driven microservice (EDM) architectures and should be avoided completely whenever possible. Eliminating cross-cutting technological and team dependencies will reduce a system’s sensitivity to change.
图 1-2显示了两种情况:左侧为单一所有权,右侧为交叉所有权。凭借唯一的所有权,该团队完全围绕两个独立的业务需求(有界上下文)进行组织,并完全控制其应用程序代码和数据库层。右侧的团队是根据技术要求组织的,其中应用程序层与数据层分开管理。这会在团队之间创建显式依赖关系,以及业务需求之间的隐式依赖关系。
Figure 1-2 shows both scenarios: sole ownership on the left and cross-cutting ownership on the right. With sole ownership, the team is fully organized around the two independent business requirements (bounded contexts) and has complete control over its application code and the database layer. On the right, the teams have been organized via technical requirements, where the application layer is managed separate from the data layer. This creates explicit dependencies between the teams, as well as implicit dependencies between the business requirements.
围绕业务需求对事件驱动的微服务架构进行建模是首选,尽管这种方法存在权衡。代码可能会被复制多次,并且许多服务可能会使用类似的数据访问模式。产品开发人员可能会尝试通过与其他产品共享数据源或通过边界耦合来减少重复。在这些情况下,从长远来看,随后的紧密耦合可能比重复逻辑和存储类似数据的成本要高得多。本书将更详细地探讨这些权衡。
Modeling event-driven microservices architectures around business requirements is preferred, though there are tradeoffs with this approach. Code may be replicated a number of times, and many services may use similar data access patterns. Product developers may try to reduce repetition by sharing data sources with other products or by coupling on boundaries. In these cases, the subsequent tight coupling may be far more costly in the long run than repeating logic and storing similar data. These tradeoffs will be examined in greater detail throughout this book.
保持有界上下文之间的松散耦合,并专注于最小化上下文间的依赖关系。这将允许有界上下文实现根据需要进行更改,而不会随后破坏许多(或任何)其他系统。
Keep loose coupling between bounded contexts, and focus on minimizing intercontext dependencies. This will allow bounded context implementations to change as necessary, without subsequently breaking many (or any) other systems.
此外,每个团队可能需要拥有完整的堆栈专业知识,这可能因需要专门的技能集和访问权限而变得复杂。组织应该实施最常见的要求,以便这些垂直团队可以支持自己,同时可以根据需要在跨团队上提供更专业的技能集。第 14 章更详细地介绍了这些最佳实践。
Additionally, each team may be required to have full stack expertise, which can be complicated by the need for specialized skill sets and access permissions. The organization should operationalize the most common requirements such that these vertical teams can support themselves, while more specialized skill sets can be provided on a cross-team, as-needed basis. These best practices are covered in more detail in Chapter 14.
组织的团队、系统和人员都必须相互沟通才能实现其目标。这些通信形成了一种互连的依赖关系拓扑,称为通信结构。共有三种主要的通信结构,每种结构都会影响企业的运营方式。
An organization’s teams, systems, and people all must communicate with one another to fulfill their goals. These communications form an interconnected topology of dependencies called a communication structure. There are three main communication structures, and each affects the way businesses operate.
业务沟通结构(图 1-3)规定了团队和部门之间的沟通,每个团队和部门都由分配给它的主要需求和职责驱动。例如,工程部门生产软件产品,销售部门向客户销售产品,支持部门确保客户和客户满意。团队的组织及其目标的规定,从主要业务单位到个人贡献者的工作,都属于这种结构。业务需求、团队分配以及团队组成都会随着时间的推移而发生变化,这会极大地影响业务沟通结构和实施沟通结构之间的关系。
The business communication structure (Figure 1-3) dictates communication between teams and departments, each driven by the major requirements and responsibilities assigned to it. For example, Engineering produces the software products, Sales sells to customers, and Support ensures that customers and clients are satisfied. The organization of teams and the provisioning of their goals, from the major business units down to the work of the individual contributor, fall under this structure. Business requirements, their assignment to teams, and team compositions all change over time, which can greatly impact the relationship between the business communication structure and the implementation communication structure.
实现通信结构(图 1-4)是与组织规定的子域模型相关的数据和逻辑。它将业务流程、数据结构和系统设计形式化,以便业务运营能够快速高效地执行。这会导致业务通信结构灵活性的折衷,因为重新定义实现必须满足的业务需求需要重写逻辑。这些重写通常是对子域模型和相关代码的迭代修改,随着时间的推移,它们反映了实现的演变,以满足新的业务需求。
The implementation communication structure (Figure 1-4) is the data and logic pertaining to the subdomain model as dictated by the organization. It formalizes business processes, data structures, and system design so that business operations can be performed quickly and efficiently. This results in a tradeoff in flexibility for the business communication structure, as redefining the business requirements that must be satisfied by the implementation requires a rewrite of the logic. These rewrites are most often iterative modifications to the subdomain model and associated code, which over time reflect the evolution of the implementation to fulfill the new business requirements.
软件工程实现通信结构的典型示例是整体数据库应用程序。应用程序的业务逻辑通过函数调用或共享状态进行内部通信。该整体应用程序又用于满足业务通信结构所规定的业务需求。
The quintessential example of an implementation communication structure for software engineering is the monolithic database application. The business logic of the application communicates internally via either function calls or shared state. This monolithic application, in turn, is used to satisfy the business requirements dictated by the business communication structure.
数据通信结构(图 1-5)是跨业务(尤其是在实现之间)进行数据通信的过程。尽管由电子邮件、即时消息和会议组成的数据通信结构经常用于传达业务变更,但它在软件实现中很大程度上被忽视了。它的作用通常是从一个系统到另一个系统临时履行的,实现通信结构通常发挥双重作用,除了其自身的要求之外还包括数据通信功能。这导致了公司如何随着时间的推移发展和变化的许多问题,其影响将在下一节中评估。
The data communication structure (Figure 1-5) is the process through which data is communicated across the business and particularly between implementations. Although a data communication structure comprising email, instant messaging, and meetings is often used for communicating business changes, it has largely been neglected for software implementations. Its role has usually been fulfilled ad hoc, from system to system, with the implementation communication structure often playing double duty by including data communication functions in addition to its own requirements. This has caused many problems in how companies grow and change over time, the impact of which is evaluated in the next section.
设计系统的组织……只能生产这些组织通信结构的副本的设计。
梅尔文·康威——委员会如何发明?(1968 年 4 月)
Organizations which design systems...are constrained to produce designs which are copies of the communication structures of these organizations.
Melvin Conway—How Do Committees Invent? (April 1968)
这句话被称为康威定律,意味着团队将根据其组织的沟通结构来构建产品。业务沟通结构将人们组织成团队,这些团队通常生产由其团队边界界定的产品。实现通信结构提供对给定产品的子域数据模型的访问,但由于数据通信能力较弱,也限制了对其他产品的访问。
This quote, known as Conway’s law, implies that a team will build products according to the communication structures of its organization. Business communication structures organize people into teams, and these teams typically produce products that are delimited by their team boundaries. Implementation communication structures provide access to the subdomain data models for a given product, but also restrict access to other products due to the weak data communication capabilities.
由于域概念跨越业务,因此组织内的其他有界上下文通常需要域数据。实现通信结构通常不擅长提供这种通信机制,尽管它们擅长满足自己的有界上下文的需求。它们以两种方式影响产品的设计。首先,由于在整个组织内沟通必要的域数据的效率低下,它们阻碍了新的、逻辑上独立的产品的创建。其次,它们提供了对现有域数据的轻松访问,但面临着不断扩展域以涵盖新业务需求的风险。这种特殊的模式通过整体设计来体现。
Because domain concepts span the business, domain data is often needed by other bounded contexts within an organization. Implementation communication structures are generally poor at providing this communication mechanism, though they excel at supplying the needs of their own bounded context. They influence the design of products in two ways. First, due to the inefficiencies of communicating the necessary domain data across the organization, they discourage the creation of new, logically separate products. Second, they provide easy access to existing domain data, at the risk of continually expanding the domain to encompass the new business requirements. This particular pattern is embodied by monolithic designs.
数据通信结构在组织设计和构建产品的过程中发挥着关键作用,但对于许多组织来说,这种结构长期以来一直缺失。如前所述,实现通信结构除了自身之外还经常扮演此角色。
Data communication structures play a pivotal role in how an organization designs and builds products, but for many organizations this structure has long been missing. As noted, implementation communication structures frequently play this role in addition to their own.
一些组织尝试缓解无法从其他实现访问域数据的问题,但这些努力都有其自身的缺点。例如,经常使用共享数据库,尽管它们经常促进反模式,并且通常无法充分扩展以适应所有性能要求。数据库可以提供只读副本;然而,这可能会不必要地暴露其内部数据模型。批处理进程可以将数据转储到文件存储以供其他进程读取,但这种方法可能会产生数据一致性和多个事实来源的问题。最后,所有这些解决方案都会导致实现之间的强耦合,并进一步将架构强化为直接的点对点 关系。
Some organizations attempt to mitigate the inability to access domain data from other implementations, but these efforts have their own drawbacks. For example, shared databases are often used, though these frequently promote anti-patterns and often cannot scale sufficiently to accommodate all performance requirements. Databases may provide read-only replicas; however, this can expose their inner data models unnecessarily. Batch processes can dump data to a file store to be read by other processes, but this approach can create issues around data consistency and multiple sources of truth. Lastly, all of these solutions result in a strong coupling between implementations and further harden an architecture into direct point-to-point relationships.
如果您发现访问组织中的数据太困难,或者由于所有数据都位于单个实施中而导致您的产品范围扩大,那么您可能会遇到不良数据通信结构的影响。随着组织的发展、新产品的开发以及越来越需要访问常用领域数据,这个问题将会被放大。
If you find that it is too hard to access data in your organization or that your products are scope-creeping because all the data is located in a single implementation, you’re likely experiencing the effects of poor data communication structures. This problem will be magnified as the organization grows, develops new products, and increasingly needs to access commonly used domain data.
组织的沟通结构极大地影响了工程实施的创建方式。在团队层面也是如此:团队的沟通结构会影响其为分配给它的特定业务需求构建的解决方案。让我们看看这在实践中是如何运作的。
An organization’s communication structures greatly influence how engineering implementations are created. This is true at the team level as well: the communication structures of the team affect the solutions it builds for the specific business requirements assigned to it. Let’s see how this works in practice.
考虑以下场景。单个团队拥有由单个数据存储支持的单个服务。他们很高兴地提供他们的业务功能,世界上一切都很好。有一天,团队负责人提出了一项新的业务需求。它与团队已经在做的事情有些相关,并且可能只是添加到他们现有的服务中。然而,它也有足够的不同,以至于它可以推出自己的新服务。
Consider the following scenario. A single team has a single service backed by a single data store. They are happily providing their business function, and all is well in the world. One day the team lead comes in with a new business requirement. It’s somewhat related to what the team is already doing and could possibly just be added on to their existing service. However, it’s also different enough that it could go into its own new service.
团队正处于十字路口:他们是在新服务中实现新业务需求,还是简单地将其添加到现有服务中?让我们更详细地看看他们的选择。
The team is at a crossroads: do they implement the new business requirement in a new service or simply add it to their existing service? Let’s take a look at their options in a bit more detail.
业务需求差异很大,因此将其放入新服务中是有意义的。但数据呢?这个新的业务功能需要一些旧数据,但这些数据目前被锁定在现有服务中。此外,该团队实际上并没有启动新的、完全独立的服务的流程。另一方面,团队变得有点大,公司发展很快。如果将来必须划分团队,那么拥有模块化和独立的系统可能会让分配所有权变得更加容易。
The business requirement is different enough that it could make sense to put it into a new service. But what about data? This new business function needs some of the old data, but that data is currently locked up in the existing service. Additionally, the team doesn’t really have a process for launching new, fully independent services. On the other hand, the team is getting to be a bit big, and the company is growing quickly. If the team has to be divided in the future, having modular and independent systems may make divvying up ownership much easier.
这种方法存在相关风险。团队必须找到一种方法从原始数据存储中获取数据并将其复制到新数据存储中。他们需要确保不会暴露内部工作原理,并且对数据结构所做的更改不会影响复制其数据的任何其他团队。此外,正在复制的数据总是有些陈旧,因为它们只能每 30 分钟实时复制一次生产数据,以免查询使数据存储饱和。需要监视和维护此连接以确保其正确运行。
There are risks associated with this approach. The team must figure out a way to source data from their original data store and copy it to their new data store. They need to ensure that they don’t expose the inner workings and that the changes they make to their data structures won’t affect any other teams copying their data. Additionally, the data being copied will always be somewhat stale, as they can only afford to copy production data in real time every 30 minutes so as not to saturate the data store with queries. This connection will need to be monitored and maintained to ensure that it is running correctly.
启动和运行新服务也存在风险。他们需要管理两个数据存储和两个服务,并为它们建立日志记录、监视、测试、部署和回滚流程。他们还必须注意同步任何数据结构更改,以免影响依赖系统。
There is also a risk in spinning up and running a new service. They will need to manage two data stores, and two services, and establish logging, monitoring, testing, deployment, and rollback processes for them. They must also take care to synchronize any data structure changes so as not to affect the dependent system.
另一种选择是在现有服务中创建新的数据结构和业务逻辑。所需的数据已在数据存储中,并且日志记录、监视、测试、部署和回滚流程已定义并正在使用。该团队熟悉该系统,并且可以立即着手实现逻辑,并且他们的整体模式支持这种服务设计方法。
The other option is to create the new data structures and business logic within the existing service. The required data is already in the data store, and the logging, monitoring, testing, deployment, and rollback processes are already defined and in use. The team is familiar with the system and can get right to work on implementing the logic, and their monolithic patterns support this approach to service design.
这种方法也存在一些风险,尽管风险比较微妙。随着更改的进行,实现中的边界可能会变得模糊,特别是因为模块通常捆绑在一起在同一代码库中。通过跨越这些边界并直接跨模块耦合来快速添加功能太容易了。快速移动有一个重大好处,但它是以紧密耦合、内聚力降低和缺乏模块化为代价的。尽管团队可以防范这种情况,但这需要出色的规划和严格遵守界限,而在面对紧张的日程、缺乏经验和转移的服务所有权时,这些界限往往会被半途而废 。
There are also risks associated with this approach, though they are a bit more subtle. Boundaries within the implementation can blur as changes are made, especially since modules are often bundled together in the same codebase. It is far too easy to quickly add features by crossing those boundaries and directly couple across the module. There is a major boon to moving quickly, but it comes at the cost of tight couplings, reduced cohesion, and a lack of modularity. Though teams can guard against this, it requires excellent planning and strict adherence to boundaries, which often fall by the wayside in the face of tight schedules, inexperience, and shifting service ownership.
大多数团队会选择第二个选项——向现有系统添加功能。这个选择并没有什么错;整体架构是有用且强大的结构,可以为企业提供非凡的价值。第一个选项首先遇到了与传统计算相关的两个问题:
Most teams would choose the second option—adding the functionality to the existing system. There is nothing wrong with this choice; monolithic architectures are useful and powerful structures and can provide exceptional value to a business. The first option runs head first into the two problems associated with traditional computing:
可靠地访问另一个系统的数据是很困难的,尤其是大规模且实时的。
Accessing another system’s data is difficult to do reliably, especially at scale and in real time.
创建和管理新服务会产生大量的开销和与之相关的风险,尤其是在组织内没有既定方法的情况下。
Creating and managing new services has substantial overhead and risk associated with it, especially if there is no established way to do so within the organization.
访问本地数据总是比访问另一个数据存储中的数据更容易。封装在另一个团队的数据存储中的任何数据都很难获取,因为它需要跨越实施和业务通信边界。随着数据、连接数量和性能要求的增长,维护和扩展变得越来越困难。
Accessing local data is always easier than accessing data in another data store. Any data encapsulated in another team’s data store is difficult to obtain, as it requires crossing both implementation and business communication boundaries. This becomes increasingly difficult to maintain and scale as data, connection count, and performance requirements grow.
尽管复制必要的数据是一种值得的方法,但它并不是万无一失的。这种模型鼓励许多直接的点对点耦合,随着组织的发展、业务部门和所有权的变化以及产品的成熟和淘汰,这些耦合变得难以维护。它对两个团队(存储数据的团队和复制数据的团队)的实现通信结构产生了严格的技术依赖性,要求他们在数据发生更改时同步工作。必须特别注意确保实现的内部数据模型不会过度暴露,以免其他系统与其紧密耦合。可扩展性、性能和系统可用性通常是两个系统的问题,因为数据复制查询可能会给源系统带来不可持续的负载。在发生紧急情况之前,可能不会注意到失败的同步过程。部落知识可能会导致团队复制数据副本,并认为这是事实的原始来源。
Though copying the necessary data over is a worthy approach, it’s not foolproof. This model encourages many direct point-to-point couplings, which become problematic to maintain as an organization grows, business units and ownership change, and products mature and phase out. It creates a strict technical dependency the implementation communication structures of both teams (the team storing the data and the team copying it), requiring them to work in synchronicity whenever a data change is made. Special care must be taken to ensure that the internal data model of an implementation is not unduly exposed, lest other systems tightly couple to it. Scalability, performance, and system availability are often issues for both systems, as the data replication query may place an unsustainable load on the source system. Failed sync processes may not be noticed until an emergency occurs. Tribal knowledge may result in a team copying a copy of data, thinking that it’s the original source of truth.
当查询完成并传输数据时,复制的数据总是有些陈旧。数据集越大、来源越复杂,副本与原始数据不同步的可能性就越大。当系统期望彼此拥有完美的、最新的副本时,尤其是在彼此就该数据进行通信时,这是有问题的。例如,由于陈旧性,报告服务可能会报告与计费服务不同的值。这可能会对服务质量、报告、分析和基于货币的决策产生严重的下游影响。
Copied data will always be somewhat stale by the time the query is complete and the data is transferred. The larger the data set and the more complex its sourcing, the more likely a copy will be out of sync with the original. This is problematic when systems expect each other to have perfect, up-to-date copies, particularly when communicating with one another about that data. For instance, a reporting service may report different values than a billing service due to staleness. This can have serious downstream consequences for service quality, reporting, analytics, and monetary-based decision making.
无法在整个公司正确传播数据并不是由于概念存在根本缺陷。恰恰相反:这是由于数据通信结构薄弱或不存在造成的。在前面的场景中,团队的实现通信结构作为极其有限的数据通信结构履行着双重职责。
The inability to correctly disseminate data throughout a company is not due to a fundamental flaw in the concept. Quite the contrary: it’s due to a weak or nonexistent data communication structure. In the preceding scenario, the team’s implementation communication structure is performing double duty as an extremely limited data communication structure.
事件驱动微服务的原则之一是核心业务数据应该易于任何需要它的服务获取和使用。这用正式的数据通信结构替换了该场景中的临时数据通信结构。对于假设的团队来说,这种数据通信结构可以消除从其他系统获取数据的大部分困难。
One of the tenets of event-driven microservices is that core business data should be easy to obtain and usable by any service that requires it. This replaces the ad hoc data communication structure in this scenario with a formalized data communication structure. For the hypothetical team, this data communication structure could eliminate most of the difficulties of obtaining data from other systems.
快进一年。该团队决定采用选项 2,并将新功能合并到同一服务中。它很快,很简单,从那时起他们已经实现了许多新功能。随着公司的发展,团队也在壮大,现在是时候重组为两个更小、更专注的团队了。
Fast-forward a year. The team decided to go with option 2 and incorporate the new features within the same service. It was quick, it was easy, and they have implemented a number of new features since then. As the company has grown, the team has grown, and now it is time for it to be reorganized into two smaller, more focused teams.
现在必须为每个新团队分配先前服务中的某些业务职能。每个团队的业务需求根据最需要关注的业务领域进行整齐划分。然而,划分实现通信结构并不容易。就像以前一样,两个团队似乎都需要大量相同的数据来满足他们的要求。出现了一系列新问题:
Each new team must now be assigned certain business functions from the previous service. The business requirements of each team are neatly divided based on areas of the business that need the most attention. Dividing the implementation communication structure, however, is not proving to be easy. Just as before, it seems that the teams both require large amounts of the same data to fulfill their requirements. New sets of questions arise:
哪个团队应该拥有哪些数据?
Which team should own which data?
数据应该驻留在哪里?
Where should the data reside?
两个团队都需要修改值的数据怎么样?
What about data where both teams need to modify the values?
团队领导决定最好只共享服务,而且他们都可以在不同的部分工作。这将需要更多的跨团队沟通和工作同步,这可能会拖累生产力。如果它们的尺寸再次增加一倍,那么将来会怎样呢?或者,如果业务需求发生了足够大的变化,以至于他们不再能够使用相同的数据结构来满足所有需求?
The team leads decide that it may be best to just share the service instead, and both of them can work on different parts. This will require a lot more cross-team communication and synchronization of efforts, which may be a drag on productivity. And what about in the future, if they double in size again? Or if the business requirements change enough that they’re no longer able to fulfill everything with the same data structure?
原团队面临着两种相互矛盾的压力。它受到影响,将所有数据保留在一项服务中,以便更快、更轻松地添加新业务功能,但代价是扩展实施通信结构。最终,团队的成长需要拆分业务沟通结构——随后将业务需求重新分配给新团队。然而,实现通信结构无法支持当前形式的重新分配,需要分解为合适的组件。这两种方法都不可扩展,并且都表明需要以不同的方式做事。这些问题都源于相同的根本原因:在实现通信结构之间通信数据的方式薄弱且定义不明确。
There are two conflicting pressures on the original team. It was influenced to keep all of its data local in one service to make adding new business functions quicker and easier, at the cost of expanding the implementation communication structure. Eventually the growth of the team necessitated splitting up the business communication structure—a requirement followed by the reassignment of business requirements to the new teams. The implementation communication structure, however, cannot support the reassignments in its current form and needs to be broken down into suitable components. Neither approach is scalable, and both point to a need to do things differently. These problems all stem from the same root cause: a weak, ill-defined means of communicating data between implementation communication structures.
事件驱动方法提供了传统的实现行为和数据通信结构的替代方案。基于事件的通信并不是请求-响应通信的直接替代,而是一种完全不同的服务之间通信方式。事件流数据通信结构将数据的生产和所有权与数据的访问分离。服务不再直接通过请求-响应 API 进行耦合,而是通过事件流中定义的事件数据进行耦合(第 3 章将详细介绍此过程)。生产者的职责仅限于将明确定义的数据生成到各自的事件流中。
The event-driven approach offers an alternative to the traditional behavior of implementation and data communication structures. Event-based communications are not a drop-in replacement for request-response communications, but rather a completely different way of communicating between services. An event-streaming data communication structure decouples the production and ownership of data from the access to it. Services no longer couple directly through a request-response API, but instead through event data defined within event streams (this process is covered more in Chapter 3). Producers’ responsibilities are limited to producing well-defined data into their respective event streams.
所有可共享的数据都发布到一组事件流,形成一个连续的、规范的叙述,详细描述组织中发生的一切。这成为系统相互通信的渠道。几乎任何事情都可以作为事件进行通信,从简单的事件到复杂的有状态记录。事件就是数据;它们不仅仅是指示数据在其他地方准备就绪的信号,或者只是从一种实现到另一种实现的直接数据传输的一种手段。相反,它们既充当数据存储又充当服务之间异步通信的手段。
All shareable data is published to a set of event streams, forming a continuous, canonical narrative detailing everything that has happened in the organization. This becomes the channel by which systems communicate with one another. Nearly anything can be communicated as an event, from simple occurrences to complex, stateful records. Events are the data; they are not merely signals indicating data is ready elsewhere or just a means of direct data transfer from one implementation to another. Rather, they act both as data storage and as a means of asynchronous communication between services.
流中的每个事件都是事实陈述,这些陈述共同构成了单一事实来源——组织内所有系统的通信基础。通信结构的好坏取决于其信息的准确性,因此组织采用事件流叙述作为单一事实来源至关重要。如果一些团队选择将冲突数据放在其他位置,则事件流作为组织数据通信主干的功能将显着减弱。
Each event in a stream is a statement of fact, and together these statements form the single source of truth—the basis of communication for all systems within the organization. A communication structure is only as good as the veracity of its information, so it’s critical that the organization adopts the event stream narrative as a single source of truth. If some teams choose instead to put conflicting data in other locations, the event stream’s function as the organization’s data communications backbone is significantly diminished.
基于事件的数据通信结构与过度扩展的实现通信结构的不同之处在于它不能提供任何查询或数据查找功能。所有业务和应用程序逻辑都必须封装在事件的生产者和消费者中。
The event-based data communication structure differs from an overextended implementation communication structure in that it is incapable of providing any querying or data lookup functionality. All business and application logic must be encapsulated within the producer and consumer of the events.
数据访问和建模需求完全转移给消费者,每个消费者从源事件流中获取自己的事件副本。任何查询复杂性也从数据所有者的实现通信结构转移到消费者的实现通信结构。消费者仍然对来自多个事件流、特殊查询功能或其他特定于业务的实现逻辑的数据的任何混合负全部责任。否则,生产者和消费者都不再需要为数据通信手段提供查询机制、数据传输机制、API(应用程序编程接口)和跨团队服务。现在,他们的责任仅限于解决其直接有限环境的需求。
Data access and modeling requirements are completely shifted to the consumer, with consumers each obtaining their own copy of events from the source event streams. Any querying complexity is also shifted from the implementation communication structure of the data owner to that of the consumer. The consumer remains fully responsible for any mixing of data from multiple event streams, special query functionality, or other business-specific implementation logic. Both producers and consumers are otherwise relieved of their duty to provide querying mechanisms, data transfer mechanisms, APIs (application programming interfaces), and cross-team services for the means of communicating data. They are now limited in their responsibility to only solving the needs of their immediate bounded context.
数据通信结构的使用是一种反转,所有可共享的数据都暴露在实现通信结构之外。并非所有数据都必须共享,因此并非所有数据都需要发布到事件流集。但是,任何其他团队或服务感兴趣的任何数据都必须发布到公共事件流集,以便数据的生产和所有权完全解耦。这提供了系统架构中长期缺失的形式化数据通信结构,并且更好地遵循松耦合和高内聚性的有界上下文原则。
The usage of a data communications structure is an inversion, with all shareable data being exposed outside of the implementation communication structure. Not all data must be shared, and thus not all of it needs to be published to the set of event streams. However, any data that is of interest to any other team or service must be published to the common set of event streams, such that the production and ownership of data becomes fully decoupled. This provides the formalized data communication structure that has long been missing from system architectures and better adheres to the bounded context principles of loose coupling and high cohesiveness.
应用程序现在可以访问通过点对点连接很难获取的数据。新服务可以简单地从规范事件流中获取任何所需的数据,创建自己的模型和状态,并执行任何必要的业务功能,而无需依赖与任何其他服务的直接点对点连接或 API。这释放了组织在任何产品中更有效地使用其大量数据的潜力,甚至以独特而强大的方式混合来自多个产品的数据。
Applications can now access data that would otherwise have been laborious to obtain via point-to-point connections. New services can simply acquire any needed data from the canonical event streams, create their own models and state, and perform any necessary business functions without depending on direct point-to-point connections or APIs with any other service. This unlocks the potential for an organization to more effectively use its vast amounts of data in any product, and even mix data from multiple products in unique and powerful ways.
事件流包含对业务运营至关重要的核心领域事件。尽管团队可能会重组,项目可能会来来去去,但重要的核心领域数据仍然可供任何需要它的新产品随时使用,而与任何特定的实施通信结构无关。这为业务提供了无与伦比的灵活性,因为对核心域事件的访问不再依赖于任何特定的实现。
Event streams contain core domain events that are central to the operation of the business. Though teams may restructure and projects may come and go, the important core domain data remains readily available to any new product that requires it, independent of any specific implementation communication structure. This gives the business unparalleled flexibility, as access to core domain events no longer relies upon any particular implementation.
事件驱动的微服务支持满足有界上下文的要求所需的业务逻辑转换和操作。这些应用程序的任务是满足这些要求并将其自己的任何必要事件发送给其他下游消费者。以下是使用事件驱动微服务的一些主要好处:
Event-driven microservices enable the business logic transformations and operations necessary to meet the requirements of the bounded context. These applications are tasked with fulfilling these requirements and emitting any of their own necessary events to other downstream consumers. Here are a few of the primary benefits of using event-driven microservices:
服务巧妙地映射到有界上下文,并且当业务需求发生变化时可以轻松重写。
Services map neatly to bounded contexts and can be easily rewritten when business requirements change.
个别服务可以根据需要扩大或缩小。
Individual services can be scaled up and down as needed.
服务使用最合适的语言和技术。这还允许使用开创性技术轻松进行原型设计。
Services use the most appropriate languages and technologies. This also allows for easy prototyping using pioneering technology.
细粒度微服务的所有权很容易重组。与大型服务相比,跨团队依赖性更少,并且组织可以更快地对业务需求的变化做出反应,否则这些变化可能会受到数据访问障碍的阻碍。
Ownership of granular microservices is easy to reorganize. There are fewer cross-team dependencies compared to large services, and the organization can react more quickly to changes in business requirements that would otherwise be impeded by barriers to data access.
事件驱动的微服务耦合在领域数据上,而不是特定的实现 API 上。数据模式可用于极大地改进数据更改的管理方式,这将在第 3 章中讨论。
Event-driven microservices are coupled on domain data and not on a specific implementation API. Data schemas can be used to greatly improve how data changes are managed, as will be discussed in Chapter 3.
可以轻松交付小型模块化微服务,并在需要时将其回滚。
It’s easy to ship a small, modular microservice, and roll it back if needed.
微服务往往比大型单体具有更少的依赖关系,从而更容易模拟所需的测试端点并确保正确的代码 覆盖率。
Microservices tend to have fewer dependencies than large monoliths, making it easier to mock out the required testing endpoints and ensure proper code coverage.
让我们回顾一下之前的团队,但具有事件驱动的数据通信结构。
Let’s revisit the team from earlier but with an event-driven data communication structure.
向团队引入了新的业务需求。它与他们当前产品的功能有些相关,但它也有足够的不同,以至于它可以进入自己的服务。将其添加到现有服务是否违反了单一责任原则并过度扩展了当前定义的有界上下文?或者它是现有服务的简单扩展,也许是添加一些新的相关数据或功能?
A new business requirement is introduced to the team. It’s somewhat related to what their current products do, but it’s also different enough that it could go into its own service. Does adding it to an existing service violate the single responsibility principle and overextend the currently defined bounded context? Or is it a simple extension, perhaps the addition of some new related data or functionality, of an existing service?
以前的技术问题(例如确定从哪里获取数据以及如何接收数据、处理批量同步问题以及实现同步 API)现在已基本消除。团队可以启动一个新的微服务,并从事件流中获取必要的数据,如果需要的话,可以一直追溯到时间的开始。团队完全有可能混合其他服务中使用的通用数据,只要该数据仅用于满足新的有界上下文的需求。这些数据的存储和结构完全由团队决定,他们可以选择保留哪些字段以及丢弃哪些字段。
Previous technical issues—such as figuring out where to source the data and how to sink it, handling batch syncing issues, and implementing synchronous APIs—are largely removed now. The team can spin up a new microservice and ingest the necessary data from the event streams, all the way back to the beginning of time if needed. It is entirely possible that the team mixes in common data used in their other services, so long as that data is used solely to fulfill the needs of the new bounded context. The storage and structure of this data are left entirely up to the team, which can choose which fields to keep and which to discard.
业务风险也得到了缓解,因为小型、更细粒度的服务允许单一团队所有权,使团队能够根据需要进行扩展和重组。当团队规模太大而无法由单个业务所有者管理时,他们可以根据需要进行拆分并重新分配微服务所有权。事件数据的所有权随着生产服务而移动,并且可以做出组织决策以减少执行未来工作所需的跨团队通信量。
Business risks are also alleviated, as the small, finer-grained services allow for single team ownership, enabling the teams to scale and reorganize as necessary. When the team grows too large to manage under a single business owner, they can split up as required and reassign the microservice ownership. The ownership of the event data moves with the producing service, and organizational decisions can be made to reduce the amount of cross-team communication required to perform future work.
只要创建新服务和获取必要数据的开销最小,微服务的本质就可以防止意大利面条式代码和庞大的单体应用占据主导地位。扩展问题现在集中在单个事件处理服务上,这些服务可以根据需要扩展其 CPU、内存、磁盘和实例计数。其余的扩展要求被转移到数据通信结构上,该结构必须确保它能够处理从事件流消费和生成事件流的各种服务负载。
The nature of the microservice prevents spaghetti code and expansive monoliths from taking hold, provided that the overhead for creating new services and obtaining the necessary data is minimal. Scaling concerns are now focused on individual event-processing services, which can scale their CPU, memory, disk, and instance count as required. The remaining scaling requirements are offloaded onto the data communication structure, which must ensure that it can handle the various loads of services consuming from and producing to its event streams.
然而,要做到这一切,团队需要确保数据确实存在于数据通信结构中,并且他们必须拥有轻松启动和管理微服务群的方法。这需要在整个组织范围内采用 EDM 架构。
To do all of this, however, the team needs to ensure that the data is indeed present in the data communication structure, and they must have the means for easily spinning up and managing a fleet of microservices. This requires an organization-wide adoption of EDM architecture.
微服务可以使用事件异步实现(本书提倡的方法),也可以同步实现,这在面向服务的架构中很常见。同步微服务往往使用请求-响应方法来实现,其中服务直接通过 API 进行通信来满足业务需求。
Microservices can be implemented asynchronously using events (the approach this book advocates) or synchronously, which is common in service-oriented architectures. Synchronous microservices tend to be fulfilled using a request-response approach, where services communicate directly through APIs to fulfill business requirements.
同步微服务存在许多问题,导致它们难以大规模使用。这并不是说公司不能通过使用同步微服务取得成功,Netflix、Lyft、Uber 和 Facebook 等公司的成就就证明了这一点。但许多公司也利用过时且混乱不堪的意大利面条式代码巨石发了财,因此不要将公司的最终成功与其底层架构的质量混为一谈。有很多书籍描述了如何实现同步微服务,因此我建议您阅读这些书籍以更好地理解同步方法。1
There are a number of issues with synchronous microservices that make them difficult to use at large scale. This is not to say that a company cannot succeed by using synchronous microservices, as evidenced by the achievements of companies such as Netflix, Lyft, Uber, and Facebook. But many companies have also made fortunes using archaic and horribly tangled spaghetti-code monoliths, so do not confuse the ultimate success of a company with the quality of its underlying architecture. There are a number of books that describe how to implement synchronous microservices, so I recommend that you read those to get a better understanding of synchronous approaches.1
此外,请注意,点对点请求响应微服务和异步事件驱动微服务都没有严格意义上的优于对方。两者在组织中都有自己的位置,因为有些任务比另一种更适合其中一种任务。然而,我自己以及许多同行和同事的经验表明,EDM 架构提供了同步请求-响应微服务所不具备的无与伦比的灵活性。也许当你读完这本书时你会同意,但至少,你会了解它们的优点和缺点。
Furthermore, note that neither point-to-point request-response microservices nor asynchronous event-driven microservices are strictly better than the other. Both have their place in an organization, as some tasks are far better suited to one over the other. However, my own experience, as well as that of many of my peers and colleagues, indicates that EDM architectures offer an unparalleled flexibility that is absent in synchronous request-response microservices. Perhaps you’ll come to agree as you proceed through this book, but at the very least, you’ll gain an understanding of their strengths and drawbacks.
以下是同步请求-响应微服务的一些最大缺点。
Here are some of the biggest shortcomings of synchronous request-response microservices.
同步微服务依赖其他服务来帮助它们执行业务任务。反过来,这些服务有它们自己的依赖服务,这些服务有它们自己的依赖服务,等等。这可能会导致过度扇出,并且难以跟踪哪些服务负责实现业务逻辑的特定部分。服务之间的连接数量可能会变得高得惊人,这进一步巩固了现有的通信结构,并使未来的改变变得更加困难。
Synchronous microservices rely on other services to help them perform their business tasks. Those services, in turn, have their own dependent services, which have their own dependent services, and so on. This can lead to excessive fanout and difficultly in tracing which services are responsible for fulfilling specific parts of the business logic. The number of connections between services can become staggeringly high, which further entrenches the existing communication structures and makes future changes more difficult.
扩展您自己的服务的能力取决于所有依赖服务的扩展能力,并且与通信扇出的程度直接相关。实施技术可能是可扩展性的瓶颈。高度可变的负载和激增的请求模式使情况变得更加复杂,所有这些都需要在整个架构中同步处理。
The ability to scale up your own service depends on the ability of all dependent services to scale up as well and is directly related to the degree of communications fanout. Implementation technologies can be a bottleneck on scalability. This is further complicated by highly variable loads and surging request patterns, which all need to be handled synchronously across the entire architecture.
如果依赖服务发生故障,则必须决定如何处理异常。生态系统中的服务越多,决定如何处理中断、何时重试、何时失败以及如何恢复以确保数据一致性变得越来越困难。
If a dependent service is down, then decisions must be made about how to handle the exception. Deciding how to handle the outages, when to retry, when to fail, and how to recover to ensure data consistency becomes increasingly difficult the more services there are within the ecosystem.
多个 API 定义和服务版本通常需要同时存在。强制客户端升级到最新的 API 并不总是可行或可取的。这可能会增加跨多个服务协调 API 更改请求的复杂性,特别是当它们伴随着底层数据结构的更改时。
Multiple API definitions and service versions will often need to exist at the same time. It is not always possible or desirable to force clients to upgrade to the newest API. This can add a lot of complexity in orchestrating API change requests across multiple services, especially if they are accompanied by changes to the underlying data structures.
在访问外部数据时,同步微服务与传统服务存在相同的问题。尽管有服务设计策略可以 减少访问外部数据的需求,但微服务通常仍然需要访问其他服务的常用数据。这将数据访问和可扩展性的责任重新放到了实现通信结构上。
Synchronous microservices have all the same problems as traditional services when it comes to accessing external data. Although there are service design strategies for mitigating the need to access external data, microservices will often still need to access commonly used data from other services. This puts the onus of data access and scalability back on the implementation communication structure.
服务可以被组合成一个分布式整体,在它们之间进行许多相互交织的调用。当团队分解整体架构并决定使用同步点对点调用来模拟整体架构内的现有边界时,通常会出现这种情况。点对点服务很容易模糊有界上下文之间的界限,因为对远程系统的函数调用可以与现有的整体代码逐行插入。
Services may be composed such that they act as a distributed monolith, with many intertwining calls being made between them. This situation often arises when a team is decomposing a monolith and decides to use synchronous point-to-point calls to mimic the existing boundaries within their monolith. Point-to-point services make it easy to blur the lines between the bounded contexts, as the function calls to remote systems can slot in line-for-line with existing monolith code.
集成测试可能很困难,因为每个服务都需要完全可操作的依赖项,而这些依赖项又需要自己的依赖项。将它们剔除可能适用于单元测试,但很少能满足更广泛的测试要求。
Integration testing can be difficult, as each service requires fully operational dependents, which require their own in turn. Stubbing them out may work for unit tests, but seldom proves sufficient for more extensive testing requirements.
同步微服务提供了许多不可否认的好处。某些数据访问模式有利于直接请求-响应耦合,例如验证用户身份和报告 AB 测试。与外部第三方解决方案的集成几乎总是使用同步机制,并且通常通过 HTTP 提供灵活的、与语言无关的通信机制。
There are a number of undeniable benefits provided by synchronous microservices. Certain data access patterns are favorable to direct request-response couplings, such as authenticating a user and reporting on an AB test. Integrations with external third-party solutions almost always use a synchronous mechanism and generally provide a flexible, language-agnostic communication mechanism over HTTP.
在同步环境中跟踪跨多个系统的操作比在异步环境中更容易。详细的日志可以显示哪些系统调用了哪些功能,从而实现了业务运营的高可调试性和可见性。
Tracing operations across multiple systems can be easier in a synchronous environment than in an asynchronous one. Detailed logs can show which functions were called on which systems, allowing for high debuggability and visibility into business operations.
托管 Web 和移动体验的服务总体上由请求响应设计提供支持,无论其同步或异步性质如何。客户会收到完全针对其需求的及时响应。
Services hosting web and mobile experiences are by and large powered by request-response designs, regardless of their synchronous or asynchronous nature. Clients receive a timely response dedicated entirely to their needs.
经验因素也非常重要,特别是当今市场上的许多开发人员往往在同步、整体式编码方面更有经验。一般来说,这使得获取同步系统人才比获取异步事件驱动开发人才更容易。
The experience factor is also quite important, especially as many developers in today’s market tend to be much more experienced with synchronous, monolithic-style coding. This makes acquiring talent for synchronous systems easier, in general, than acquiring talent for asynchronous event-driven development.
公司的架构很少(如果有的话)完全基于事件驱动的微服务。混合架构肯定会成为常态,其中同步和异步解决方案根据问题空间的需要并排部署。
A company’s architecture could only rarely, if ever, be based entirely on event-driven microservices. Hybrid architectures will certainly be the norm, where synchronous and asynchronous solutions are deployed side-by-side as the problem space requires.
通信结构指导软件在组织生命周期中的创建和管理方式。数据通信结构通常不发达并且是临时的,但是引入持久的、易于访问的域事件集(如事件驱动系统所体现的那样)可以使用更小的、专门构建的实现。
Communication structures direct how software is created and managed through the life of an organization. Data communication structures are often underdeveloped and ad hoc, but the introduction of a durable, easy-to-access set of domain events, as embodied by event-driven systems, enables smaller, purpose-built implementations to be used.
1例如,请参阅Sam Newman 的《构建微服务》 (O'Reilly,2015 年)以及Kasun Indrasiri 和 Prabath Siriwardena 的《企业微服务》 (Apress,2018 年) 。
1 See, for example, Building Microservices by Sam Newman (O’Reilly, 2015) and Microservices for the Enterprise by Kasun Indrasiri and Prabath Siriwardena (Apress, 2018).
事件驱动的微服务是为了满足特定的有界上下文而构建的小型应用程序。消费者微服务消费并处理来自一个或多个输入事件流的事件,而生产者微服务将事件生成到事件流以供其他服务使用。事件驱动的微服务通常是一组输入事件流的使用者和另一组输出事件流的生产者。这些服务可能是无状态的(参见第 5 章)或有状态的(参见第 7 章),还可能包含同步请求-响应 API(参见第 13 章)。这些服务都共享从事件代理消费事件或向事件代理生成事件的通用功能。事件驱动的微服务之间的通信是完全异步的。
An event-driven microservice is a small application built to fulfill a specific bounded context. Consumer microservices consume and process events from one or more input event streams, whereas producer microservices produce events to event streams for other services to consume. It is common for an event-driven microservice to be a consumer of one set of input event streams and a producer of another set of output event streams. These services may be stateless (see Chapter 5) or stateful (see Chapter 7) and may also contain synchronous request-response APIs (see Chapter 13). These services all share the common functionality of consuming events from or producing events to the event broker. Communication between event-driven microservices is completely asynchronous.
事件流由事件代理提供服务,本章后半部分将更详细地介绍事件代理。以任何有意义的规模运行微服务通常需要使用部署管道和容器管理系统,这也在本章末尾处进行了讨论。
Event streams are served by an event broker, which is covered in more detail in the second half of this chapter. Running microservices at any meaningful scale often necessitates using deployment pipelines and container management systems, also discussed near the end of this chapter.
在事件驱动的微服务的讨论中经常出现术语“拓扑” 。该术语通常用于表示单个微服务的处理逻辑。它还可用于指各个微服务、事件流和请求-响应 API 之间的类似图形的关系。让我们依次看看每个定义。
The term topology comes up frequently in discussions of event-driven microservices. This term is often used to mean the processing logic of an individual microservice. It may also be used to refer to the graph-like relationship between individual microservices, event streams, and request-response APIs. Let’s look at each definition in turn.
微服务拓扑是单个微服务内部的事件驱动拓扑。这定义了要对传入事件执行的数据驱动操作,包括转换、存储和发射。
A microservice topology is the event-driven topology internal to a single microservice. This defines the data-driven operations to be performed on incoming events, including transformation, storage, and emission.
图 2-1显示了从两个输入事件流摄取的单个微服务拓扑。
Figure 2-1 shows a single microservice topology ingesting from two input event streams.
微服务拓扑从事件流 A 中摄取事件并将其具体化到数据存储中。本章稍后将更详细地介绍具体化操作。同时,事件流 B 被摄取,某些事件被过滤、转换,然后与存储的状态结合起来。结果输出到新的事件流。微服务的摄取、处理和输出是其拓扑的一部分。
The microservice topology ingests events from event stream A and materializes them into a data store. The materialization operation is covered in greater detail later in this chapter. Meanwhile, event stream B is ingested, and certain events are filtered out, transformed, and then joined against the stored state. The results are output to a new event stream. The ingestion, processing, and output of the microservice are part of its topology.
业务拓扑是一组实现复杂业务功能的微服务、事件流和 API。它是任意的服务分组,可能代表单个团队或部门拥有的服务,也可能代表实现复杂业务功能超集的服务。第 1 章中详细介绍的业务通信结构构成了业务拓扑。微服务实现业务有界上下文,事件流提供共享跨上下文域数据的数据通信机制。
A business topology is the set of microservices, event streams, and APIs that fulfill complex business functions. It is an arbitrary grouping of services and may represent the services owned by a single team or department or those that fulfill a superset of complex business functionality. The business communication structures detailed in Chapter 1 compose the business topology. Microservices implement the business bounded contexts, and event streams provide the data communication mechanism for sharing cross-context domain data.
微服务拓扑详细描述了单个微服务的内部工作原理。另一方面,业务拓扑详细描述了服务之间的关系。
A microservice topology details the inner workings of a single microservice. A business topology, on the other hand, details the relationships between services.
图 2-2显示了具有三个独立的微服务和事件流的业务拓扑。请注意,业务拓扑并未详细说明微服务的内部工作原理。
Figure 2-2 shows a business topology with three independent microservices and event streams. Note that the business topology does not detail the inner workings of a microservice.
微服务 1 消费并转换事件流 A 中的数据,并将结果生成到事件流 B。微服务 2 和微服务 3 都消费事件流 B 中的数据。微服务 2 严格充当消费者,并提供 REST API,可以在其中同步访问数据。同时,微服务3根据其有界上下文要求执行自己的转换,并输出到事件流C。新的微服务和事件流可以根据需要添加到业务拓扑中,通过事件流异步耦合。
Microservice 1 consumes and transforms data from event stream A and produces the results to event stream B. Microservice 2 and microservice 3 both consume from event stream B. Microservice 2 acts strictly as a consumer and provides a REST API in which data can be accessed synchronously. Meanwhile, microservice 3 performs its own transformations according to its bounded context requirements and outputs to event stream C. New microservices and event streams can be added to the business topology as needed, coupled asynchronously through event streams.
事件可以是业务通信结构范围内发生的任何事情。收到发票、预订会议室、请求一杯咖啡(是的,您可以将咖啡机连接到事件流)、雇用新员工以及成功完成任意代码都是企业内发生的事件的示例。重要的是要认识到事件可以是对业务重要的任何事物。一旦开始捕获这些事件,就可以创建事件驱动的系统来在整个组织中利用和使用它们。
An event can be anything that has happened within the scope of the business communication structure. Receiving an invoice, booking a meeting room, requesting a cup of coffee (yes, you can hook a coffee machine up to an event stream), hiring a new employee, and successfully completing arbitrary code are all examples of events that happen within a business. It is important to recognize that events can be anything that is important to the business. Once these events start being captured, event-driven systems can be created to harness and use them across the organization.
事件是对所发生事件的记录,就像应用程序的信息和错误日志记录应用程序中发生的事件一样。然而,与这些日志不同的是,事件也是唯一的事实来源,如第 1 章所述。因此,它们必须包含准确描述所发生事件所需的所有信息。
An event is a recording of what happened, much like how an application’s information and error logs record what takes place in the application. Unlike these logs, however, events are also the single source of truth, as covered in Chapter 1. As such, they must contain all the information required to accurately describe what happened.
事件通常使用键/值格式表示。value 存储事件的完整详细信息,而 key 用于对具有相同 key 的事件进行识别、路由和聚合操作。该键不是所有事件类型的必填字段。
Events are typically represented using a key/value format. The value stores the complete details of the event, while the key is used for identification purposes, routing, and aggregation operations on events with the same key. The key is not a required field for all event types.
| 钥匙 | 价值 |
|---|---|
唯一身份 Unique ID |
与唯一 ID 相关的详细信息 Details pertaining to the Unique ID |
本书将使用三种主要事件类型,并且您将在自己的领域中不可避免地遇到它们。
There are three main event types, which will be used throughout this book and which you’ll inevitably encounter in your own domains.
无键事件用于将事件描述为单一的事实陈述。一个示例可以是指示客户与产品交互的事件,例如用户在数字图书平台上打开图书实体。顾名思义,该事件不涉及任何密钥。
Unkeyed events are used to describe an event as a singular statement of fact. An example could be an event indicating that a customer interacted with a product, such as a user opening a book entity on a digital book platform. As the name implies, there is no key involved in this event.
| 钥匙 | 价值 |
|---|---|
不适用 N/A |
ISBN:372719,时间戳:1538913600 ISBN: 372719, Timestamp: 1538913600 |
实体是唯一的事物,并且以该事物的唯一 ID 为关键。实体事件描述实体(最常见的是业务上下文中的对象)在给定时间点的属性和状态。对于图书出版商来说,一个示例可以是使用 ISBN 键入的图书实体。值字段包含与唯一实体相关的所有必要信息。
An entity is a unique thing and is keyed on the unique ID of that thing. The entity event describes the properties and state of an entity—most commonly an object in the business context—at a given point in time. For a book publisher, an example could be a book entity, keyed on ISBN. The value field contains all the necessary information related to the unique entity.
| 钥匙 | 价值 |
|---|---|
国际标准书号:372719 ISBN: 372719 |
作者:亚当·贝尔马尔 Author: Adam Bellemare |
实体事件在事件驱动架构中尤其重要。它们提供实体状态的连续历史记录,并可用于具体化状态(在下一节中介绍)。只需要最新的实体事件即可确定实体的当前状态。
Entity events are particularly important in event-driven architectures. They provide a continual history of the state of an entity and can be used to materialize state (covered in the next section). Only the latest entity event is needed to determine the current state of an entity.
键控事件包含键,但不代表实体。键控事件通常用于对事件流进行分区,以保证事件流的单个分区内的数据局部性(本章稍后将详细介绍)。一个例子是事件流,以 ISBN 为键,指示哪个用户与该书进行了交互。
A keyed event contains a key but does not represent an entity. Keyed events are usually used for partitioning the stream of events to guarantee data locality within a single partition of an event stream (more on this later in the chapter). An example could be a stream of events, keyed on ISBN, indicating which user has interacted with the book.
| 钥匙 | 价值 |
|---|---|
国际标准书号:372719 ISBN: 372719 |
用户 ID:A537FE UserId: A537FE |
国际标准书号:372719 ISBN: 372719 |
用户ID:BB0012 UserId: BB0012 |
请注意,事件可以按键聚合,以便可以为每个 ISBN 组成用户列表,从而产生以ISBN 为键的单个实体事件。
Note that the events could be aggregated by key such that a list of users can be composed for each ISBN, resulting in a single entity event keyed on ISBN..
您可以通过按顺序应用实体事件流中的实体事件来具体化有状态表。每个实体事件都被更新插入到键/值表中,以便表示给定键的最近读取的事件。相反,您可以通过将每个更新发布到事件流来将表转换为实体事件流。这称为表流二元性,它是事件驱动微服务中状态创建的基础。如图 2-3所示,其中 AA 和 CC 的物化表中都有最新值。
You can materialize a stateful table by applying entity events, in order, from an entity event stream. Each entity event is upserted into the key/value table, such that the most recently read event for a given key is represented. Conversely, you can convert a table into a stream of entity events by publishing each update to the event stream. This is known as the table-stream duality, and it is fundamental to the creation of state in an event-driven microservice. This is illustrated in Figure 2-3, where AA and CC both have the newest values in their materialized table.
更新插入意味着如果表中尚不存在新行则插入新行,如果存在则更新它。
Upserting means inserting a new row if it doesn’t already exist in the table, or updating it if it does.
以同样的方式,您可以让表记录所有更新,并在此过程中生成表示表随时间变化的状态的数据流。在以下示例中,BB 被更新插入两次,而 DD 仅被更新插入一次。图 2-4中的输出流显示了代表这些操作的三个 upsert 事件。
In the same way, you can have a table record all updates and in doing so produce a stream of data representing the table’s state over time. In the following example, BB is upserted twice, while DD is upserted just once. The output stream in Figure 2-4 shows three upsert events representing these operations.
例如,关系数据库表是通过一系列数据插入、更新和删除命令创建和填充的。这些命令可以作为不可变日志的事件生成,例如本地仅附加文件(如 MySQL 中的二进制日志)或外部事件流。通过回放日志的全部内容,您可以准确地重建表及其所有数据内容。
A relational database table, for instance, is created and populated through a series of data insertion, update, and deletion commands. These commands can be produced as events to an immutable log, such as a local append-only file (like the binary log in MySQL) or an external event stream. By playing back the entire contents of the log, you can exactly reconstruct the table and all of its data contents.
这种表流二元性用于在事件驱动的微服务之间传递状态。任何消费者客户端都可以读取键控事件的事件流并将其具体化到自己的本地状态存储中。这种简单而强大的模式允许微服务仅通过事件共享状态,而生产者和消费者服务之间没有任何直接耦合。
This table-stream duality is used for communicating state between event-driven microservices. Any consumer client can read an event stream of keyed events and materialize it into its own local state store. This simple yet powerful pattern allows microservices to share state through events alone, without any direct coupling between producer and consumer services.
通过生成逻辑删除来处理键控事件的删除。逻辑删除是一个键控事件,其值设置为 null。这是一种约定,向消费者表明具有该键的事件应该从物化数据存储中删除,因为上游生产者已声明它现在已被删除。
The deletion of a keyed event is handled by producing a tombstone. A tombstone is a keyed event with its value set to null. This is a convention that indicates to the consumer that the event with that key should be removed from the materialized data store, as the upstream producer has declared that it is now deleted.
仅追加的不可变日志可能会无限增长,除非它们被压缩。事件代理执行压缩,通过仅保留给定键的最新事件来减少其内部日志的大小。具有相同键的旧事件将被删除,剩余事件将被压缩到一组新的、更小的文件中。维护事件流偏移量,以便消费者不需要进行任何更改。图 2-5说明了事件代理中事件流的逻辑压缩,包括逻辑删除记录的完全删除。
Append-only immutable logs may grow indefinitely unless they are compacted. Compaction is performed by the event broker to reduce the size of its internal logs by retaining only the most recent event for a given key. Older events of the same key will be deleted, and the remaining events compacted down into a new and smaller set of files. The event stream offsets are maintained such that no changes are required by the consumers. Figure 2-5 illustrates the logical compaction of an event stream in the event broker, including the total deletion of the tombstone record.
压缩减少了磁盘使用量和达到当前状态必须处理的事件数量,但代价是消除了事件流提供的事件历史记录。
Compaction reduces both disk usage and the number of events that must be processed to reach the current state, at the expense of eliminating the history of events otherwise provided by the event stream.
维护业务逻辑处理的状态是事件驱动架构中极其常见的模式。几乎可以肯定的是,您的整个业务模型将无法适应纯粹的无状态流域,因为过去的业务决策将影响您今天做出的决策。举一个具体的例子,如果您的业务是零售业,您将需要知道您的库存水平,以确定何时需要重新订购,并避免向客户出售您没有的商品。您还希望能够跟踪您的应付账款和应收账款。也许您希望每周向所有向您提供电子邮件地址的客户发送促销信息。所有这些系统都要求您能够将事件流具体化为当前状态表示。
Maintaining state for the processing of business logic is an extremely common pattern in an event-driven architecture. It is a near-certainty that your entire business model will not be able to fit in a purely stateless streaming domain, as past business decisions will influence decisions you make today. As a specific example, if your business is retail, you will need to know your stock level to identify when you need to reorder and to avoid selling customers items you do not have. You also want to be able to keep track of your accounts payable and accounts receivable. Perhaps you want to have a weekly promotion sent to all the customers who have provided you their email addresses. All of these systems require that you have the ability to materialize streams of events into current state representations.
事件数据充当长期和与实现无关的数据存储的手段,以及服务之间的通信机制。因此,事件的生产者和消费者对数据的含义有共同的理解是很重要的。理想情况下,消费者必须能够解释事件的内容和含义,而无需咨询生产服务的所有者。这需要一种用于生产者和消费者之间通信的通用语言,类似于同步请求-响应服务之间的 API 定义。
Event data serves as the means of long term and implementation agnostic data storage, as well as the communication mechanism between services. Therefore, it is important that both the producers and consumers of events have a common understanding of the meaning of the data. Ideally, the consumer must be able to interpret the contents and meaning of an event without having to consult with the owner of the producing service. This requires a common language for communication between producers and consumers and is analogous to an API definition between synchronous request-response services.
Apache Avro和Google 的 Protobuf等架构选择提供了在事件驱动的微服务中大量利用的两个功能。首先,它们提供了一个演化框架,可以安全地对模式进行某些更改,而不需要下游消费者进行代码更改。其次,它们还提供了生成类型化类(如果适用)的方法,以将模式化数据转换为您选择的语言的普通旧对象。这使得微服务开发中业务逻辑的创建变得更加简单和透明。第 3 章更详细地介绍了这些主题。
Schematization selections such as Apache Avro and Google’s Protobuf provide two features that are leveraged heavily in event-driven microservices. First, they provide an evolution framework, where certain sets of changes can be safely made to the schemas without requiring downstream consumers to make a code change. Second, they also provide the means to generate typed classes (where applicable) to convert the schematized data into plain old objects in the language of your choice. This makes the creation of business logic far simpler and more transparent in the development of microservices. Chapter 3 covers these topics in greater detail.
每个事件流都有一个且唯一一个生产微服务。该微服务是该流生成的每个事件的所有者。通过允许通过系统跟踪数据沿袭,这样就可以始终了解任何给定事件的权威事实来源。正如第 14 章所讨论的,访问控制机制应该用于强制所有权和写入边界。
Each event stream has one and only one producing microservice. This microservice is the owner of each event produced to that stream. This allows for the authoritative source of truth to always be known for any given event, by permitting the tracing of data lineage through the system. Access control mechanisms, as discussed in Chapter 14, should be used to enforce ownership and write boundaries.
每个生产就绪的事件驱动型微服务平台的核心都是事件代理。这是一个接收事件、将它们存储在队列或分区事件流中并提供它们供其他进程使用的系统。事件通常根据其底层逻辑含义发布到不同的流,类似于数据库如何拥有许多表,每个表在逻辑上分开以包含特定类型的数据。
At the heart of every production-ready event-driven microservice platform is the event broker. This is a system that receives events, stores them in a queue or partitioned event stream, and provides them for consumption by other processes. Events are typically published to different streams based on their underlying logical meaning, similar to how a database will have many tables, each logically separated to contain a specific type of data.
适用于大型企业的事件代理系统通常都遵循相同的模型。多个分布式事件代理在集群中协同工作,为事件流的生成和消费提供平台。该模型提供了大规模运行事件驱动的生态系统所需的几个基本功能:
Event broker systems suitable for large-scale enterprises all generally follow the same model. Multiple, distributed event brokers work together in a cluster to provide a platform for the production and consumption of event streams. This model provides several essential features that are required for running an event-driven ecosystem at scale:
可以添加额外的事件代理实例来增加集群的生产、消费和数据存储容量。
Additional event broker instances can be added to increase the cluster’s production, consumption, and data storage capacity.
事件数据在节点之间复制。这允许代理集群在代理发生故障时保留并继续提供数据。
Event data is replicated between nodes. This permits a cluster of brokers to both preserve and continue serving data when a broker fails.
事件代理节点集群使客户端能够在代理发生故障时连接到其他节点。这使得客户能够保持完整的正常运行时间。
A cluster of event broker nodes enables clients to connect to other nodes in the case of a broker failure. This permits the clients to maintain full uptime.
多个broker节点分担生产和消费负载。此外,每个代理节点必须具有高性能,才能每秒处理数十万次写入或读取。
Multiple broker nodes share the production and consumption load. In addition, each broker node must be highly performant to be able to handle hundreds of thousands of writes or reads per second.
尽管在事件代理的幕后存储、复制和访问事件数据的方式有多种,但它们通常都提供相同的存储和访问机制给客户端。
Though there are different ways in which event data can be stored, replicated, and accessed behind the scenes of an event broker, they all generally provide the same mechanisms of storage and access to their clients.
These are the minimal requirements of the underlying storage of the data by the broker:
事件流可以分为单独的子流,子流的数量可以根据生产者和消费者的需求而变化。这种分区机制允许消费者的多个实例并行处理每个子流,从而实现更大的吞吐量。请注意,队列不需要分区,尽管出于性能目的无论如何对它们进行分区可能很有用。
Event streams can be partitioned into individual substreams, the number of which can vary depending on the needs of the producer and consumer. This partitioning mechanism allows for multiple instances of a consumer to process each substream in parallel, allowing for far greater throughput. Note that queues do not require partitioning, though it may be useful to partition them anyway for performance purposes.
事件流分区中的数据是严格排序的,并且按照与最初发布的顺序完全相同的顺序提供给客户端。
Data in an event stream partition is strictly ordered, and it is served to clients in the exact same order that it was originally published.
所有事件数据一旦发布就完全不可变。没有任何机制可以在事件数据发布后对其进行修改。您只能通过发布包含更新数据的新事件来更改以前的数据。
All event data is completely immutable once published. There is no mechanism that can modify event data once it is published. You can alter previous data only by publishing a new event with the updated data.
事件在写入事件流时会被分配一个索引。消费者使用它来管理数据消耗,因为他们可以指定从哪个偏移量开始读取。消费者当前指数与尾部指数之间的差异就是消费者滞后。该指标可用于在消费者数量较高时扩大消费者数量,在消费者数量较低时减少消费者数量。此外,它还可以用于唤醒功能即服务逻辑。
Events are assigned an index when written to the event stream. This is used by the consumers to manage data consumption, as they can specify which offset to begin reading from. The difference between the consumer’s current index and the tail index is the consumer lag. This metric can be used to scale up the number of consumers when it is high, and scale them down when it is low. Additionally, it can also be used to awaken Functions-as-a-Service logic.
事件流必须能够无限期地保留事件。此属性是维护事件流中的状态的基础。
Event streams must be able to retain events for an infinite period of time. This property is foundational for maintaining state in an event stream.
事件流必须是可重播的,以便任何消费者都可以读取所需的任何数据。这为单一事实来源提供了基础,也是微服务之间通信状态的基础。
Event streams must be replayable, such that any consumer can read whatever data it requires. This provides the basis for the single source of truth and is foundational for communicating state between microservices.
选择事件代理时还需要考虑许多其他因素。
There are a number of additional factors to consider in the selection of an event broker.
支持工具对于有效开发事件驱动的微服务至关重要。其中许多工具都与事件代理本身的实现绑定在一起。其中一些包括:
Support tools are essential for effectively developing event-driven microservices. Many of these tools are bound to the implementation of the event broker itself. Some of these include:
浏览事件和模式数据
Browsing of event and schema data
配额、访问控制和主题管理
Quotas, access control, and topic management
监控、吞吐量和滞后测量
Monitoring, throughput, and lag measurements
有关您可能需要的工具的更多信息,请参阅第 14 章。
See Chapter 14 for more information regarding tooling you may need.
Hosted services allow you to outsource the creation and management of your event broker.
是否存在托管解决方案?
Do hosted solutions exist?
您会购买托管解决方案还是内部托管?
Will you purchase a hosted solution or host it internally?
托管代理是否提供监控、扩展、灾难恢复、复制和多区域部署?
Does the hosting agent provide monitoring, scaling, disaster recovery, replication, and multizone deployments?
它会将您与单个特定服务提供商联系起来吗?
Does it couple you to a single specific service provider?
是否有专业的支持服务?
Are there professional support services available?
有多种事件代理实现可供选择,每种实现都有不同级别的客户端支持。您常用的语言和工具与客户端库良好配合非常重要。
There are multiple event broker implementations to select from, each of which has varying levels of client support. It is important that your commonly used languages and tools work well with the client libraries.
是否存在所需语言的客户端库和框架?
Do client libraries and frameworks exist in the required languages?
如果这些库不存在,您能够构建它们吗?
Will you be able to build the libraries if they do not exist?
您正在使用常用的框架还是尝试推出自己的框架?
Are you using commonly used frameworks or trying to roll your own?
社区支持是选择事件经纪人的一个极其重要的方面。开源且免费的项目(例如Apache Kafka)是具有大型社区支持的事件代理的一个特别好的示例。
Community support is an extremely important aspect of selecting an event broker. An open source and freely available project, such as Apache Kafka, is a particularly good example of an event broker with large community support.
有在线社区支持吗?
Is there online community support?
技术是否成熟并已做好量产准备?
Is the technology mature and production-ready?
该技术是否在许多组织中得到普遍使用?
Is the technology commonly used across many organizations?
该技术对未来员工有吸引力吗?
Is the technology attractive to prospective employees?
员工会对使用这些技术进行构建感到兴奋吗?
Will employees be excited to build with these technologies?
根据事件流的大小和保留持续时间,最好将较旧的数据段存储在速度较慢但较便宜的存储中。分层存储提供多层访问性能,事件代理或其数据服务节点本地的专用磁盘提供最高性能层。后续层可以包括诸如专用大规模存储层服务(例如,亚马逊的S3、谷歌云存储和Azure 存储)之类的选项。
Depending on the size of your event streams and the duration of retention, it may be preferable to store older data segments in slower but cheaper storage. Tiered storage provides multiple layers of access performance, with a dedicated disk local to the event broker or its data-serving nodes providing the highest performance tier. Subsequent tiers can include options such as dedicated large-scale storage layer services (e.g., Amazon’s S3, Google Cloud Storage, and Azure Storage).
是否自动支持分层存储?
Is tiered storage automatically supported?
数据可以根据使用情况滚动到较低或较高的层吗?
Can data be rolled into lower or higher tiers based on usage?
数据可以从存储在哪一层中无缝检索吗?
Can data be seamlessly retrieved from whichever tier it is stored in?
我发现人们可能对消息代理和事件代理的构成感到困惑。事件代理可以代替消息代理,但消息代理无法实现事件代理的所有功能。让我们更深入地比较它们。
I have found that people may be confused about what constitutes a message broker and what constitutes an event broker. Event brokers can be used in place of a message broker, but a message broker cannot fulfill all the functions of an event broker. Let’s compare them in more depth.
消息代理有着悠久的历史,并已被众多组织用于大规模的面向消息的中间件架构中。消息代理使系统能够通过发布/订阅消息队列在网络上进行通信。生产者将消息写入队列,而消费者则消费这些消息并进行相应的处理。然后,消息被确认为已使用,并立即或不久后删除。消息代理旨在处理与事件代理不同类型的问题。
Message brokers have a long history and have been used in large-scale message-oriented middleware architectures by numerous organizations. Message brokers enable systems to communicate across a network through publish/subscribe message queues. Producers write messages to a queue, while a consumer consumes these messages and processes them accordingly. Messages are then acknowledged as consumed and deleted either immediately or shortly thereafter. Message brokers are designed to handle a different type of problem than event brokers.
另一方面,事件代理是围绕提供有序的事实日志而设计的。事件代理可以满足消息代理无法满足的两个非常具体的需求。其一,消息代理仅提供消息 队列,其中消费消息的处理是基于每个队列的。共享队列消耗的应用程序将仅接收记录的子集。这使得不可能通过事件正确地传达状态,因为每个消费者无法获得所有事件的完整副本。与消息代理不同,事件代理维护单个记录分类帐并通过索引管理单独的访问,因此每个独立的使用者都可以访问所有所需的事件。此外,消息代理会在确认后删除事件,而事件代理会根据组织需要保留事件。消费后删除事件使得消息代理不足以为所有应用程序提供无限期存储的、全局可访问的、可重播的单一事实来源。
Event brokers, on the other hand, are designed around providing an ordered log of facts. Event brokers meet two very specific needs that are not satisfied by the message broker. For one, the message broker provides only queues of messages, where the consumption of the message is handled on a per-queue basis. Applications that share consumption from a queue will each receive only a subset of the records. This makes it impossible to correctly communicate state via events, since each consumer is unable to obtain a full copy of all events. Unlike the message broker, the event broker maintains a single ledger of records and manages individual access via indices, so each independent consumer can access all required events. Additionally, a message broker deletes events after acknowledgment, whereas an event broker retains them for as long as the organization needs. The deletion of the event after consumption makes a message broker insufficient for providing the indefinitely stored, globally accessible, replayable, single source of truth for all applications.
事件代理支持不可变的、仅附加的事实日志,以保留事件排序的状态。消费者可以随时从日志中的任何位置拾取并重新处理。此模式对于启用事件驱动的微服务至关重要,但它不适用于消息代理。
Event brokers enable an immutable, append-only log of facts that preserves the state of event ordering. The consumer can pick up and reprocess from anywhere in the log at any time. This pattern is essential for enabling event-driven microservices, but it is not available with message brokers.
请记住,消息代理中使用的队列在事件驱动的微服务中仍然发挥着作用。队列提供了有用的访问模式,但在严格分区的事件流中实现起来可能很困难。消息代理系统引入的模式当然是 EDM 架构的有效模式,但它们不足以满足此类架构所需的全部职责。本书的其余部分不关注任何消息代理架构或应用程序设计,而是关注事件代理在 EDM 架构中的使用。
Keep in mind that queues, as used in message brokers, still have a role in event-driven microservices. Queues provide useful access patterns that may be awkward to implement with strictly partitioned event streams. The patterns introduced by message broker systems are certainly valid patterns for EDM architectures, but they are not sufficient for the full scope of duties such architectures require. The remainder of the book focuses not on any message broker architectures or application design, but instead on the usage of event brokers in EDM architectures.
虽然不是明确的标准,但常用的事件代理使用仅附加的不可变日志。事件附加在日志末尾,并给出自动递增索引 ID。数据的使用者使用对索引 ID 的引用来访问数据。然后,事件可以作为事件流或队列使用,具体取决于业务需求和事件代理的可用功能。
Though not a definitive standard, commonly available event brokers use an append-only immutable log. Events are appended at the end of the log and given an autoincrementing index ID. Consumers of the data use a reference to the index ID to access data. Events can then be consumed as either an event stream or a queue, depending on the needs of the business and the available functionality of the event broker.
每个使用者负责将其自己的指针更新为事件流中先前读取的索引。该索引称为offset,是从事件流开头开始的当前事件的测量值。偏移量允许多个消费者相互独立地消费和跟踪他们的进度,如图2-6所示。
Each consumer is responsible for updating its own pointers to previously read indices within the event stream. This index, known as the offset, is the measurement of the current event from the beginning of the event stream. Offsets permit multiple consumers to consume and track their progress independently of one another, as shown in Figure 2-6.
消费者组允许将多个消费者视为同一逻辑实体,并可用于消息消费的水平扩展。新消费者加入消费者组,导致事件流分区分配重新分配。新消费者仅使用其分配的分区中的事件,就像组中先前的旧消费者实例继续仅使用其剩余分配的分区中的事件一样。通过这种方式,可以在同一消费者组之间平衡事件消费,同时确保给定分区的所有事件都由单个消费者实例独占消费。组中活动消费者实例的数量仅限于事件流中分区的数量。
The consumer group allows for multiple consumers to be viewed as the same logical entity and can be leveraged for horizontal scaling of message consumption. A new consumer joins the consumer group, causing a redistribution of event stream partition assignments. The new consumer consumes events only from its assigned partitions, just as older consumer instances previously in the group continue to consume only from their remaining assigned partitions. In this way, event consumption can be balanced across the same consumer group, while ensuring that all events for a given partition are exclusively consumed by a single consumer instance. The number of active consumer instances in the group is limited to the number of partitions in the event stream.
在基于队列的消费中,每个事件由一个且仅一个微服务实例消费。被消费后,该事件被事件代理标记为“已消费”,并且不再提供给任何其他消费者。作为队列消费时,分区计数并不重要,因为可以使用任意数量的消费者实例进行消费。
In queue-based consumption, each event is consumed by one and only one microservice instance. Upon being consumed, that event is marked as “consumed” by the event broker and is no longer provided to any other consumer. Partition counts do not matter when consuming as a queue, as any number of consumer instances can be used for consumption.
从队列进行处理时,不会维持事件顺序。并行消费者无序地消费和处理事件,而单个消费者可能无法处理某个事件,将其返回到队列以便稍后处理,然后继续处理下一个事件。
Event order is not maintained when processing from a queue. Parallel consumers consume and process events out of order, while a single consumer may fail to process an event, return it to the queue for processing at a later date, and move on to the next event.
并非所有事件代理都支持队列。例如,Apache Pulsar目前支持队列,而阿帕奇卡夫卡没有。图 2-7显示了使用单独偏移确认的队列的实现。
Queues are not supported by all event brokers. For instance, Apache Pulsar currently supports queues while Apache Kafka does not. Figure 2-7 shows the implementation of a queue using individual offset acknowledgment.
持久且不可变的日志为单一事实来源提供了存储机制,事件代理成为服务消费和生成数据的唯一位置。这样,就可以保证每个消费者都获得相同的数据副本。
The durable and immutable log provides the storage mechanism for the single source of truth, with the event broker becoming the only location in which services consume and produce data. This way, every consumer is guaranteed to be given an identical copy of the data.
采用事件代理作为单一事实来源需要组织中的文化转变。以前,团队可能只是编写直接 SQL 查询来访问整体数据库中的数据,而现在团队还必须将整体数据发布到事件代理。管理单体应用的开发人员必须确保生成的数据完全准确,因为事件流和单体数据库之间的任何分歧都将被视为生产团队的失败。数据的使用者不再直接耦合在整体上,而是直接从事件流中消费。
Adopting the event broker as the single source of truth requires a culture shift in the organization. Whereas previously a team may simply have written direct SQL queries to access data in a monolith’s database, now the team must also publish the monolith’s data to the event broker. The developers managing the monolith must ensure that the data produced is fully accurate, because any disagreement between the event streams and the monolith’s database will be considered a failure of the producing team. Consumers of the data no longer couple directly on the monolith, but instead consume directly from the event streams.
采用事件驱动的微服务可以创建仅使用事件代理来存储和访问数据的服务。虽然微服务的业务逻辑肯定会使用事件的本地副本,但事件代理仍然是所有数据的单一事实来源。
The adoption of event-driven microservices enables the creation of services that use only the event broker to store and access data. While local copies of the events may certainly be used by the business logic of the microservice, the event broker remains the single source of truth for all data.
随着服务数量的增加,管理微服务可能变得越来越困难。每个微服务都需要特定的计算资源、数据存储、配置、环境变量以及一整套其他特定于微服务的属性。每个微服务还必须可由拥有它的团队管理和部署。容器化和虚拟化及其相关的管理系统是实现这一目标的常见方法。这两个选项都允许各个团队通过单个可部署单元来定制其微服务的需求 。
Managing microservices can become increasingly difficult as the number of services grows. Each microservice requires specific compute resources, data stores, configurations, environment variables, and a whole host of other microservice-specific properties. Each microservice must also be manageable and deployable by the team that owns it. Containerization and virtualization, along with their associated management systems, are common ways to achieve this. Both options allow individual teams to customize the requirements of their microservices through a single unit of deployability.
最近由Docker普及的容器将应用程序彼此隔离。容器通过共享内核模型利用现有的主机操作系统。这提供了容器之间的基本分离,而容器本身隔离了环境变量、库和其他依赖项。容器以一小部分成本提供了虚拟机的大部分优势(接下来将介绍),并且启动时间短且资源开销低。
Containers, as recently popularized by Docker, isolate applications from one another. Containers leverage the existing host operating system via a shared kernel model. This provides basic separation between containers, while the container itself isolates environment variables, libraries, and other dependencies. Containers provide most of the benefits of a virtual machine (covered next) at a fraction of the cost, with fast startup times and low resource overhead.
容器的共享操作系统方法确实有一些权衡。容器化应用程序必须能够在主机操作系统上运行。如果应用程序需要专门的操作系统,则需要设置独立的主机。安全性是主要问题之一,因为容器共享对主机操作系统的访问。内核中的漏洞可能会使该主机上的所有容器面临风险。对于友好的工作负载,这不太可能成为问题,但云计算中当前的共享租赁模型开始使其成为一个更大的考虑因素。
Containers’ shared operating system approach does have some tradeoffs. Containerized applications must be able to run on the host OS. If an application requires a specialized OS, then an independent host will need to be set up. Security is one of the major concerns, since containers share access to the host machine’s OS. A vulnerability in the kernel can put all the containers on that host at risk. With friendly workloads this is unlikely to be a problem, but current shared tenancy models in cloud computing are beginning to make it a bigger consideration.
虚拟机 (VM) 解决了容器的一些缺点,尽管其采用速度较慢。传统虚拟机通过为每个实例指定的独立操作系统和虚拟化硬件提供完全隔离。尽管这种替代方案提供了比容器更高的安全性,但从历史上看它的成本要高得多。与容器相比,每个虚拟机的开销成本更高,启动时间更慢,系统占用空间更大。
Virtual machines (VMs) address some of the shortcomings of containers, though their adoption has been slower. Traditional VMs provide full isolation with a self-contained OS and virtualized hardware specified for each instance. Although this alternative provides higher security than containers, it has historically been much more expensive. Each VM has higher overhead costs compared to containers, with slower startup times and larger system footprints.
人们正在努力使虚拟机变得更便宜、更高效。目前的举措包括谷歌的 gVisor、亚马逊的 Firecracker和Kata Containers,仅举几例。随着这些技术的改进,虚拟机将成为容器的更具竞争力的替代品,以满足您的微服务需求。如果您的需求是由安全第一的需求驱动的,那么值得关注这个领域。
Efforts are under way to make VMs cheaper and more efficient. Current initiatives include Google’s gVisor, Amazon’s Firecracker, and Kata Containers, to mention just a few. As these technologies improve, VMs will become a much more competitive alternative to containers for your microservice needs. It is worth keeping an eye on this domain should your needs be driven by security-first requirements.
容器和虚拟机通过各种称为容器管理系统(CMS) 的专用软件进行管理。它们控制容器部署、资源分配以及与底层计算资源的集成。流行且常用的 CMS 包括 Kubernetes、Docker Engine、Mesos Marathon、Amazon ECS和Nomad。
Containers and VMs are managed through a variety of purpose-built software known as container management systems (CMSes). These control container deployment, resource allocation, and integration with the underlying compute resources. Popular and commonly used CMSes include Kubernetes, Docker Engine, Mesos Marathon, Amazon ECS, and Nomad.
微服务必须能够根据不断变化的工作负载、服务级别协议 (SLA) 和性能要求进行扩展和缩减。必须支持垂直扩展,即在每个微服务实例上增加或减少 CPU、内存和磁盘等计算资源。还必须支持水平扩展,添加或删除新实例。
Microservices must be able to scale up and down depending on changing workloads, service-level agreements (SLAs), and performance requirements. Vertical scaling must be supported, in which compute resources such as CPU, memory, and disk are increased or decreased on each microservice instance. Horizontal scaling must also be supported, with new instances added or removed.
每个微服务应作为单个单元部署。对于许多微服务来说,执行其业务需求只需要一个可执行文件,并且可以将其部署在单个容器中。其他微服务可能更复杂,需要协调多个容器和外部数据存储。这就是 Kubernetes 的 Pod 概念发挥作用的地方,它允许通过单个操作部署和恢复多个容器。Kubernetes 还允许单次运行操作;例如,数据库迁移可以在单个可部署的执行期间运行。
Each microservice should be deployed as a single unit. For many microservices, a single executable is all that is needed to perform its business requirements, and it can be deployed within a single container. Other microservices may be more complex, with multiple containers and external data stores requiring coordination. This is where something like Kubernetes’s pod concept comes into play, allowing for multiple containers to be deployed and reverted as a single action. Kubernetes also allows for single-run operations; for example, database migrations can be run during the execution of the single deployable.
许多实现都支持虚拟机管理,但目前比容器管理受到更多限制。Kubernetes 和 Docker Engine 支持 Google 的 gVisor 和 Kata Containers,而亚马逊的平台则支持 AWS Firecracker。随着开发的继续,容器和虚拟机之间的界限将继续模糊。确保您选择的 CMS 能够处理您需要的容器和虚拟机。
VM management is supported by a number of implementations, but is currently more limited than container management. Kubernetes and Docker Engine support Google’s gVisor and Kata Containers, while Amazon’s platform supports AWS Firecracker. The lines between containers and VMs will continue to blur as development continues. Make sure that the CMS you select will handle the containers and VMs that you require of it.
有丰富的资源可用于Kubernetes、Docker、Mesos、Amazon ECS和Nomad。他们提供的信息远远超出了我在这里所能提供的范围。我鼓励您查看这些材料以获取更多信息。
There are rich sets of resources available for Kubernetes, Docker, Mesos, Amazon ECS, and Nomad. The information they provide goes far beyond what I can present here. I encourage you to look into these materials for more information.
微服务税是与实施微服务架构的工具和组件相关的成本总和,包括财务、人力和机会。这包括管理、部署和操作事件代理、CMS、部署管道、监控解决方案和日志记录服务的成本。这些费用是不可避免的,要么由组织集中支付,要么由每个实施微服务的团队独立支付。前者导致了用于开发微服务的可扩展、简化和统一的框架,而后者导致了过多的开销、重复的解决方案、碎片化的工具和不可持续的增长。
The microservice tax is the sum of costs, including financial, manpower, and opportunity, associated with implementing the tools and components of a microservice architecture. This includes the cost of managing, deploying, and operating the event broker, CMS, deployment pipelines, monitoring solutions, and logging services. These expenses are unavoidable and are paid either centrally by the organization or independently by each team implementing microservices. The former results in a scalable, simplified, and unified framework for developing microservices, while the latter results in excessive overhead, duplicate solutions, fragmented tooling, and unsustainable growth.
缴纳微服务税并不是一件小事,它是开始使用 EDM 的最大障碍之一。小型组织可能会最好坚持使用更适合其业务需求的架构,例如模块化整体架构。较大的组织需要考虑 微服务平台的实施和维护的总成本,并确定其业务的长期路线图是否能够适应预测的工作量。
Paying the microservice tax is not a trivial matter, and it is one of the largest impediments to getting started with EDM. Small organizations would likely do best to stick with an architecture that better suits their business needs, such as a modular monolith. Larger organizations need to account for the total costs of both the implementation and maintenance of a microservice platform and determine if the long-term roadmap of their business can accommodate the predicted work efforts.
幸运的是,近年来,开源和托管服务都变得更加可用且易于使用。随着 CMS、事件代理和其他常用工具之间的新集成,微服务税正在稳步降低。确保您的组织准备好投入必要的资源来支付这些前期成本。
Fortunately, both open source and hosted services have become far more available and easy to use in recent years. The microservice tax is being steadily reduced with new integrations between CMSes, event brokers, and other commonly needed tools. Be sure that your organization is prepared to devote the necessary resources to pay these up-front costs.
本章涵盖了事件驱动的微服务背后的基本要求。事件代理是数据通信的主要机制,提供大规模的实时事件流供其他服务使用。容器化和容器管理系统允许大规模运行微服务。最后,本章还介绍了事件和事件驱动逻辑的重要原则,并初步介绍了分布式事件驱动世界中的状态管理。
This chapter covered the basic requirements behind event-driven microservices. The event broker is the main mechanism of data communication, providing real-time event streams at scale for other services to consume. Containerization and container management systems allow for running microservices at scale. Lastly, this chapter also covered the important principles underlying events and event-driven logic and provided a first look at managing state in an distributed event-driven world.
通信的基本问题是在一个点精确地或近似地再现在另一点选择的消息。
克劳德·香农
The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point.
Claude Shannon
被称为信息论之父的香农指出了沟通的最大障碍:确保消息的消费者能够准确地再现生产者的消息,从而正确地传达内容和含义。生产者和消费者必须对消息有共同的理解;否则,可能会被误解,沟通也不完整。在事件驱动的生态系统中,事件是消息和通信的基本单位。事件必须尽可能准确地描述发生的情况及其原因。它是对事实的陈述,当与系统中的所有其他事件相结合时,提供了所发生事件的完整历史记录。
Shannon, known as the Father of Information Theory, identified the largest hurdle of communication: ensuring that a consumer of a message can accurately reproduce the producer’s message, such that both the content and meaning are correctly conveyed. The producer and consumer must have a common understanding of the message; otherwise, it may be misinterpreted, and the communication will be incomplete. In the event-driven ecosystem, the event is the message and the fundamental unit of communication. An event must describe as accurately as possible what happened and why. It is a statement of fact and, when combined with all the other events in a system, provides a complete history of what has happened.
要通信的数据的格式和创建数据的逻辑构成了数据契约。事件数据的生产者和消费者都遵循该契约。它赋予事件超出其生成上下文的含义和形式,并将数据的可用性扩展到消费者应用程序。
The format of the data to be communicated and the logic under which it is created form the data contract. This contract is followed by both the producer and the consumer of the event data. It gives the event meaning and form beyond the context in which it is produced and extends the usability of the data to consumer applications.
定义良好的数据契约有两个组成部分。首先是数据定义,或者将生成什么(即字段、类型和各种数据结构)。第二个组件是触发逻辑,或者说它产生的原因(即触发事件创建的特定业务逻辑)。随着业务需求的发展,可以对数据定义和触发逻辑进行更改。
There are two components of a well-defined data contract. First is the data definition, or what will be produced (i.e., the fields, types, and various data structures). The second component is the triggering logic, or why it is produced (i.e., the specific business logic that triggered the event’s creation). Changes can be made to both the data definition and the triggering logic as the business requirements evolve.
更改数据定义时必须小心,以免删除或更改下游消费者正在使用的字段。同样,修改触发逻辑时也必须小心。更改数据定义比更改触发机制更为常见,因为更改后者通常会破坏原始事件定义的含义。
You must take care when changing the data definition, so as not to delete or alter fields that are being used by downstream consumers. Similarly, you must also be careful when modifying the triggering logic. It is far more common to change the data definition than the triggering mechanism, as altering the latter often breaks the meaning of the original event definition.
强制执行数据契约并提供一致性的最佳方法是为每个事件定义一个架构。生产者定义了一个显式模式,详细说明了数据定义和触发逻辑,同一类型的所有事件都遵循此格式。在此过程中,生产者提供了一种将其事件格式传达给所有潜在消费者的机制。反过来,消费者可以自信地根据模式化数据构建他们的微服务业务逻辑。
The best way to enforce data contracts and provide consistency is to define a schema for each event. The producer defines an explicit schema detailing the data definition and the triggering logic, with all events of the same type adhering to this format. In doing so, the producer provides a mechanism for communicating its event format to all prospective consumers. The consumers, in turn, can confidently build their microservice business logic against the schematized data.
缺乏显式预定义模式的生产者和消费者之间基于事件的通信的任何实现都将不可避免地最终依赖于隐式模式。隐式数据合同很脆弱,容易受到不受控制的变化,这可能会给下游消费者带来很多不必要的困难。
Any implementation of event-based communication between a producer and consumer that lacks an explicit predefined schema will inevitably end up relying on an implicit schema. Implicit data contracts are brittle and susceptible to uncontrolled change, which can cause much undue hardship to downstream consumers.
消费者必须能够提取其业务流程所需的数据,如果没有对哪些数据应该可用的一系列期望,它就无法做到这一点。消费者通常必须依靠部落知识和团队间沟通来解决数据问题,随着事件流和团队数量的增加,这个过程无法扩展。要求每个消费者独立解释数据也存在很大的风险,因为消费者对数据的解释可能与同行不同,这会导致对单一事实来源的看法不一致。
A consumer must be able to extract the data necessary for its business processes, and it cannot do that without having a set of expectations about what data should be available. Consumers must often rely on tribal knowledge and interteam communication to resolve data issues, a process that is not scalable as the number of event streams and teams increases. There is also substantial risk in requiring each consumer to independently interpret the data, as a consumer may interpret it differently than its peers, which leads to inconsistent views of the single source of truth.
构建一个通用库来解释所有服务的任何给定事件可能很诱人,但这会带来多种语言格式、事件演变和独立发布周期的问题。跨服务重复工作以确保隐式定义数据的一致视图非常重要,最好完全避免。
It may be tempting to build a common library that interprets any given event for all services, but this creates problems with multiple language formats, event evolutions, and independent release cycles. Duplicating efforts across services to ensure a consistent view of implicitly defined data is nontrivial and best avoided completely.
生产者在隐式模式方面也处于劣势。即使有最好的意图,生产者也可能没有注意到(或者他们的单元测试没有揭示)他们已经更改了事件数据定义。如果没有明确检查其服务的事件格式,这种情况可能会被忽视,直到导致下游消费者失败。显式模式为消费者和生产者提供安全性和稳定性。
Producers are also at a disadvantage with implicit schemas. Even with the best of intentions, a producer may not notice (or perhaps their unit tests don’t reveal) that they have altered their event data definition. Without an explicit check of their service’s event format, this situation may go unnoticed until it causes downstream consumers to fail. Explicit schemas give security and stability to both consumers and producers.
支持模式定义中的集成注释和任意元数据对于传达事件的含义至关重要。围绕事件的产生和消费的知识应尽可能接近事件定义。模式注释有助于消除数据含义的歧义并减少消费者误解的可能性。有两个主要领域的评论特别有价值:
Support for integrated comments and arbitrary metadata in the schema definition is essential for communicating the meaning of an event. The knowledge surrounding the production and consumption of events should be kept as close as possible to the event definition. Schema comments help remove ambiguity about the data’s meaning and reduce the chance of misinterpretation by consumers. There are two main areas where comments are particularly valuable:
Specifying the triggering logic of the event. This is typically done in a block header at the top of the schema definition and should clearly state why an event has been generated.
提供有关结构化模式中特定字段的上下文和清晰度。例如,日期时间字段的注释可以指定格式是 UTC、ISO 还是 Unix 时间。
Giving context and clarity about a particular field within the structured schema. For example, a datetime field’s comments could specify if the format is UTC, ISO, or Unix time.
模式格式必须支持全方位的模式演化规则。模式演变使生产者能够更新其服务的输出格式,同时允许消费者继续不间断地消费事件。业务变更可能需要添加新字段、弃用旧字段或扩展字段范围。模式演化框架确保这些更改可以安全地发生,并且生产者和消费者可以彼此独立地更新。
The schema format must support a full range of schema evolution rules. Schema evolution enables producers to update their service’s output format while allowing consumers to continue consuming the events uninterrupted. Business changes may require that new fields be added, old fields be deprecated, or the scope of a field be expanded. A schema evolution framework ensures that these changes can occur safely and that producers and consumers can be updated independently of one another.
如果没有模式演化支持,服务的更新将变得极其昂贵。生产者和消费者被迫密切协调,以前兼容的旧数据可能不再与当前系统兼容。每当生产者更改数据模式时,期望消费者更新其服务是不合理的。事实上,微服务的一个核心价值主张是,除了特殊情况外,它们应该独立于其他服务的发布周期。
Updates to services become prohibitively expensive without schema evolution support. Producers and consumers are forced to coordinate closely, and old, previously compatible data may no longer be compatible with current systems. It is unreasonable to expect consumers to update their services whenever a producer changes the data schema. In fact, a core value proposition of microservices is that they should be independent of the release cycles of other services except in exceptional cases.
一组明确的模式演化规则对于使消费者和生产者能够及时更新其应用程序大有帮助。这些规则称为兼容性类型。
An explicit set of schema evolution rules goes a long way in enabling both consumers and producers to update their applications in their own time. These rules are known as compatibility types.
允许读取使用较新模式生成的数据,就像使用较旧模式生成的数据一样。在事件驱动的架构中,这是一个特别有用的演化需求,因为最常见的系统更改模式始于生产者更新其数据定义并使用较新的模式生成数据。如果使用者需要访问新字段,则只需更新其架构和代码的副本。
Allows for data produced with a newer schema to be read as though it were produced with an older schema. This is a particularly useful evolutionary requirement in an event-driven architecture, as the most common pattern of system change begins with the producer updating its data definition and producing data with the newer schema. The consumer is required only to update its copy of the schema and code should it need access to the new fields.
允许读取使用较旧模式生成的数据,就好像它是使用较新模式生成的一样。这使得数据使用者能够使用较新的模式来读取较旧的数据。在以下几种情况下,此功能特别有用:
消费者期待上游团队提供新功能。如果新的模式已经定义,消费者可以在生产者发布之前发布自己的更新。
架构编码数据由部署在客户硬件上的产品发送,例如报告用户指标的手机应用程序。可以对新生产者版本的模式格式进行更新,同时保持与以前版本的兼容性。
消费者应用程序可能需要重新处理使用旧模式版本生成的事件流中的数据。模式演变确保消费者可以将其转换为熟悉的版本。如果不遵循向后兼容性,消费者将只能读取最新格式的消息。
Allows for data produced with an older schema to be read as though it were produced with a newer schema. This enables a consumer of data to use a newer schema to read older data. There are several scenarios where this is particularly useful:
The consumer is expecting a new feature to be delivered by the upstream team. If the new schema is already defined, the consumer can release its own update prior to the producer release.
Schema-encoded data is sent by a product deployed on customer hardware, such as a cell phone application that reports on user metrics. Updates can be made to the schema format for new producer releases, while maintaining the compatibility with previous releases.
The consumer application may need to reprocess data in the event stream that was produced with an older schema version. Schema evolution ensures that the consumer can translate it to a familiar version. If backward compatibility is not followed, the consumer will only be able to read messages with the latest format.
向前兼容性和向后兼容性的结合,这是最有力的保证,也是您应该尽可能使用的保证。您始终可以在以后放宽兼容性要求,但收紧它们通常要困难得多。
The union of forward compatibility and backward compatibility, this is the strongest guarantee and the one you should use whenever possible. You can always loosen the compatibility requirements at a later date, but it is often far more difficult to tighten them.
代码生成器用于将事件模式转换为给定编程语言的类定义或等效结构。生产者使用此类定义来创建和填充新的事件对象。编译器或序列化器(取决于实现)要求生成器尊重数据类型并填充原始模式中指定的所有不可为空字段。然后,生产者创建的对象将转换为其序列化格式并发送到事件代理,如图3-1所示。
A code generator is used to turn an event schema into a class definition or equivalent structure for a given programming language. This class definition is used by the producer to create and populate new event objects. The producer is required by the compiler or serializer (depending on the implementation) to respect data types and populate all non-nullable fields that are specified in the original schema. The objects created by the producer are then converted into their serialized format and sent to the event broker, as shown in Figure 3-1.
事件数据的使用者维护自己的模式版本,该版本通常与生产者的版本相同,但可能是较旧或较新的模式,具体取决于模式演化的使用。如果观察到完全兼容性,则服务可以使用任何版本的架构来生成其定义。使用者读取事件并使用其编码的架构版本对其进行反序列化。事件格式要么与消息一起存储,这在规模上可能非常昂贵,要么存储在架构注册表中并按需访问(请参阅“架构注册表”)。一旦反序列化为其原始格式,事件就可以转换为使用者支持的模式版本。此时,演化规则开始发挥作用,默认值应用于缺失的字段,而未使用的字段则完全丢弃。最后,根据模式生成的类将数据转换为对象。此时,消费者的业务逻辑就可以开始运行了。该流程如图3-2所示。
The consumer of the event data maintains its own version of the schema, which is often the same version as the producer’s but could be an older or newer schema, depending on the usage of schema evolution. If full compatibility is being observed, the service can use any version of the schema to generate its definitions. The consumer reads the event and deserializes it using the schema version that it was encoded with. The event format is stored either alongside the message, which can be prohibitively expensive at scale, or in a schema registry and accessed on-demand (see “Schema Registry”). Once deserialized into its original format, the event can be converted to the version of the schema supported by the consumer. Evolution rules come into play at this point, with defaults being applied to missing fields, and unused fields dropped completely. Finally, the data is converted into an object based on the schema-generated class. At this point, the consumer’s business logic may begin its operations. This process is shown in Figure 3-2.
代码生成器支持的最大好处是能够使用您选择的语言根据类定义编写应用程序。如果您使用的是编译语言,代码生成器会提供编译器检查,以确保您不会错误处理事件类型或丢失任何给定的非事件类型。null数据字段。除非遵守架构,否则您的代码将无法编译,因此如果不遵守架构数据定义,您的应用程序将不会被发布。编译语言和非编译语言都受益于有一个类实现来进行编码。当您尝试将错误的类型传递给构造函数或设置器时,现代 IDE 会通知您,而如果您使用通用格式(例如对象键/值映射),您将不会收到任何通知。降低数据处理不当的风险可以使整个生态系统的数据质量更加一致
。
The biggest benefit of code generator support is being able to write your application against a class definition in the language of your choice. If you are using a compiled language, the code generator provides compiler checks to ensure that you aren’t mishandling event types or missing the population of any given non-null data field. Your code will not compile unless it adheres to the schema, and therefore your application will not be shipped without adhering to the schema data definition. Both compiled and noncompiled languages benefit from having a class implementation to code against. A modern IDE will notify you when you’re trying to pass the wrong types into a constructor or setter, whereas you would receive no notification if you’re instead using a generic format such as an object key/value map. Reducing the risk of mishandling data provides for much more consistent data quality across the
ecosystem.
有时,架构定义必须以某种方式进行更改,从而导致破坏性的演化变化。发生这种情况的原因有很多,包括不断变化的业务需求改变了原始域的模型、原始域的范围不正确以及定义模式时的人为错误。虽然可以相当容易地更改生产服务以适应新模式,但对下游消费者的影响各不相同,需要考虑在内。
There are times when the schema definition must change in a way that results in a breaking evolutionary change. This can happen for a number of reasons, including evolving business requirements that alter the model of the original domain, improper scoping of the original domain, and human error while defining the schema. While the producing service can be fairly easily changed to accommodate the new schema, the impacts to downstream consumers vary and need to be taken into account.
在处理破坏性架构更改时,最重要的事情是尽早与下游消费者进行清晰的沟通。确保任何迁移计划得到所有相关人员的理解和批准,并且没有人措手不及。
The most important thing when dealing with breaking schema changes is to communicate early and clearly with downstream consumers. Ensure that any migration plans have the understanding and approval of everyone involved and that no one is caught unprepared.
虽然要求生产者和消费者之间进行密切协调似乎有些严厉,但数据合同的重新协商和领域模型的改变需要每个人的支持。除了重新协商架构之外,您还需要采取一些额外的步骤来适应新架构和从中创建的新事件流。破坏性架构更改往往对于无限期存在的实体影响很大,但对于给定时间段后过期的事件影响较小。
While it may seem heavy-handed to require intense coordination between producers and consumers, the renegotiation of the data contract and the alteration of the domain model require buy-in from everyone. Aside from renegotiating the schema, you need to take some additional steps to accommodate the new schema and new event streams that are created from it. Breaking schema changes tend to be quite impactful for entities that exist indefinitely, but less so for events that expire after a given period of time.
对实体模式的重大更改相当罕见,因为这种情况通常需要重新定义原始域模型,从而无法简单地扩展当前模型。新实体将在新模式下创建,而以前的实体是在旧模式下生成的。数据定义的这种差异给您带来了两种选择:
Breaking changes to an entity schema are fairly rare, as this circumstance typically requires a redefinition of the original domain model such that the current model cannot simply be extended. New entities will be created under the new schema, while previous entities were generated under the old schema. This divergence of data definition leaves you with two choices:
与新旧模式相抗衡。
Contend with both the old and new schemas.
以新模式格式重新创建所有实体(通过迁移或从源重新创建它们)。
Re-create all entities in the new schema format (via migration, or by re-creating them from source).
第一个选项对于生产者来说是最简单的,但它只是将不同实体定义的解析推给了消费者。这与减少消费者单独解释数据的需求的目标相矛盾,并增加了误解的风险、服务之间处理不一致以及维护系统的复杂性显着提高。
The first option is the easiest for the producer, but it simply pushes off the resolution of the different entity definitions onto the consumer. This contradicts the goal of reducing the need for consumers to interpret the data individually and increases the risk of misinterpretation, inconsistent processing between services, and significantly higher complexity in maintaining systems.
现实情况是,在解决不同的模式定义方面,消费者永远不会比生产者处于更好的位置。将这一责任推给消费者是不好的做法。
The reality is that the consumer will never be in a better position than the producer for resolving divergent schema definitions. It is bad practice to defer this responsibility to the consumer.
第二种选择对于生产者来说更困难,但可以确保新旧业务实体一致地重新定义。在实践中,生产者必须重新处理导致生成旧实体的源数据,并应用新的业务逻辑以新格式重新创建实体。这种方法迫使组织作为一个整体来解决这些实体的含义以及生产者和消费者应如何理解和使用它们。
The second option is more difficult for the producer, but ensures that the business entities, both old and new, are redefined consistently. In practice, the producer must reprocess the source data that led to the generation of the old entities and apply the new business logic to re-create the entities under the new format. This approach forces the organization as a whole to resolve what these entities mean and how they should be understood and used by producer and consumer alike.
将旧实体保留在原始事件流中的旧模式下,因为您可能需要它们来重新处理验证和取证调查。使用新模式将新实体和更新实体生成到新流。
Leave the old entities under the old schema in their original event stream, because you may need them for reprocessing validation and forensic investigations. Produce the new and updated entities using the new schema to a new stream.
当您合并重大更改时,非实体事件往往更容易处理。最简单的选项是创建一个新的事件流并开始向该流生成新事件。必须通知旧流的使用者,以便他们可以将自己注册为新事件流的使用者。每个消费服务还必须考虑两个事件定义之间业务逻辑的差异。
Nonentity events tend to be simpler to deal with when you are incorporating breaking changes. The simplest option is to create a new event stream and begin producing the new events to that stream. The consumers of the old stream must be notified so that they can register themselves as consumers of the new event stream. Each consuming service must also account for the divergence in business logic between the two event definitions.
不要在事件流中混合不同的事件类型,尤其是进化上不兼容的事件类型。事件流开销很便宜,逻辑分离对于确保消费者在处理他们需要处理的事件时拥有完整的信息和明确的定义非常重要。
Don’t mix different event types in an event stream, especially event types that are evolutionarily incompatible. Event stream overhead is cheap, and the logical separation is important in ensuring that consumers have full information and explicit definitions when dealing with the events they need to process.
鉴于旧事件流不再产生新事件,每个消费服务的使用者最终将赶上最新的记录。随着时间的推移,流的保留期最终将导致流的完全清除,此时所有使用者都可以注销自己,并且可以删除事件流。
Given that the old event stream no longer has new events being produced to it, the consumers of each consuming service will eventually catch up to the latest record. As time goes on, the stream’s retention period will eventually result in a full purging of the stream, at which point all consumers can unregister themselves and the event stream can be deleted.
虽然有许多选项可用于格式化和序列化事件数据,但数据契约最好使用强定义的格式(例如 Avro、Thrift 或 Protobuf)来实现。一些最流行的事件代理框架支持序列化和反序列化使用这些格式编码的事件。例如,Apache Kafka和Apache Pulsar都支持 JSON、Protobuf 和 Avro 架构格式。支持这两种技术的机制是模式注册表, “模式注册表”中有更详细的介绍。尽管对这些序列化选项的详细评估和比较超出了本书的范围,但有许多可用的在线资源可以帮助您在这些特定选项中做出决定。
While there are many options available for formatting and serializing event data, data contracts are best fulfilled with strongly defined formats such as Avro, Thrift, or Protobuf. Some of the most popular event broker frameworks have support for serializing and deserializing events encoded with these formats. For example, both Apache Kafka and Apache Pulsar support JSON, Protobuf, and Avro schema formats. The mechanism of support for both of the technologies is the schema registry, which is covered in more detail in “Schema Registry”. Though a detailed evaluation and comparison of these serialization options is beyond the scope of this book, there are a number of online resources available that can help you decide among these particular options.
您可能会想选择使用简单键/值对的纯文本事件形式的更灵活的选项,它仍然提供一些结构,但不提供显式模式或模式演化框架。然而,请谨慎使用这种方法,因为它可能会损害微服务通过强大的数据契约保持彼此隔离的能力,从而需要更多的团队间通信。
You may be tempted to choose a more flexible option in the form of plain-text events using simple key/value pairs, which still offers some structure but provides no explicit schema or schema evolution frameworks. Proceed cautiously with this approach, however, as it can compromise microservices’ ability to remain isolated from one another via a strong data contract, requiring far more interteam communication.
非结构化纯文本事件通常会成为生产者和消费者的负担,特别是当用例和数据随着时间的推移而变化时。如前所述,我建议选择一种强定义的显式模式格式,支持受控模式演化,例如 Apache Avro 或 Protobuf。我不推荐 JSON,因为它不提供完全兼容的模式演化。
Unstructured plain-text events usually become a burden to both the producer and the consumer, particularly as use cases and data changes over time. As mentioned, I recommend instead choosing a strongly defined, explicit schema format that supports controlled schema evolution, such as Apache Avro or Protobuf. I do not recommend JSON, as it does not provide full-compatibility schema evolution.
创建事件定义时需要遵循许多最佳实践,并且需要避免一些反模式。请记住,随着事件驱动的微服务支持的架构数量的增加,事件定义的数量也在增加。精心设计的活动将最大限度地减少消费者和生产者的重复性痛点。话虽如此,以下规则都不是硬性规定。您可以按照自己认为合适的方式打破它们,但我建议您在继续之前仔细考虑其影响的全部范围以及问题空间的权衡。
There are a number of best practices to follow when you are creating event definitions, as well as several anti-patterns to avoid. Keep in mind that as the number of architectures powered by event-driven microservices expands, so does the number of event definitions. Well-designed events will minimize the otherwise repetitive pain points for both consumers and producers. With that being said, none of the following are hard-and-fast rules. You can break them as you see fit, though I recommend that you think very carefully about the full scope of implications and the tradeoffs for your problem space before proceeding.
一个好的事件定义不仅仅是一条指示发生某事的消息,而是对该事件期间发生的所有事情的完整描述。用业务术语来说,这是摄取输入数据并应用业务逻辑时生成的结果数据。该输出事件必须被视为单一事实来源,并且必须被记录为不可变的事实以供下游消费者消费。它对实际发生的事情拥有完全的权威,消费者不需要查阅任何其他数据源就知道发生了这样的事件。
A good event definition is not simply a message indicating that something happened, but rather the complete description of everything that happened during that event. In business terms, this is the resultant data that is produced when input data is ingested and the business logic is applied. This output event must be treated as the single source of truth and must be recorded as an immutable fact for consumption by downstream consumers. It is the full and total authority on what actually occurred, and consumers should not need to consult any other source of data to know that such an event took place.
事件流应包含代表单个逻辑事件的事件。不建议在事件流中混合不同类型的事件,因为这样做可能会混淆事件是什么以及流代表什么的定义。验证正在生成的模式很困难,因为在这种情况下可能会动态添加新模式。尽管在特殊情况下您可能希望忽略此原则,但在架构工作流程中生成和使用的绝大多数事件流都应具有严格的单一定义 。
An event stream should contain events representing a single logical event. It is not advisable to mix different types of events within an event stream, because doing so can muddle the definitions of what the event is and what the stream represents. It is difficult to validate the schemas being produced, as new schemas may be added dynamically in such a scenario. Though there are special circumstances where you may wish to ignore this principle, the vast majority of event streams produced and consumed within your architectural workflow should each have a strict, single definition.
对事件数据使用最窄的类型。这使您可以依靠代码生成器、语言类型检查(如果支持)和序列化单元测试来检查数据的边界。这听起来很简单,但在很多情况下,当您不使用正确的类型时,就会出现歧义。以下是一些容易避免的现实例子:
Use the narrowest types for your event data. This lets you rely on the code generators, language type checking (if supported), and serialization unit tests to check the boundaries of your data. It sounds simple, but there are many cases where ambiguity can creep in when you don’t use the proper types. Here are a few easily avoidable real-world examples:
string存储数值string to store a numeric value这需要消费者解析字符串并将其转换为数值,并且通常会得出 GPS 坐标。这很容易出错并且容易失败,尤其是在null发送值或空字符串时。
This requires the consumer to parse and convert the string to a numeric value and often comes up with GPS coordinates. This is error prone and subject to failures, especially when a null value or an empty string is sent.
integer布尔值integer as a boolean而0和1分别可以用来表示 false 和 true,什么2意思呢?怎么样-1?
While 0 and 1 can be used to denote false and true, respectively, what does 2 mean? How about -1?
string枚举string as an enum这对于生产者来说是有问题的,因为他们必须确保其发布的值与可接受的伪枚举列表相匹配。拼写错误和不正确的值将不可避免地被引入。对此字段感兴趣的消费者需要知道可能值的范围,这将需要与生产团队交谈,除非在模式的注释中指定。无论哪种情况,这都是隐式定义,因为生产者无法防止字符串中值范围的任何更改。整个方法根本就是不好的做法。
This is problematic for producers, as they must ensure that their published values match an accepted pseudo-enum list. Typos and incorrect values will inevitably be introduced. A consumer interested in this field will need to know the range of possible values, and this will require talking to the producing team, unless it’s specified in the comments of the schema. In either case, this is an implicit definition, since the producers are not guarded against any changes to the range of values in the string. This whole approach is simply bad practice.
通常会避免使用枚举,因为生产者担心创建消费者模式中不存在的新枚举标记。但是,消费者有责任考虑它无法识别的枚举令牌,并确定是否应该使用默认值处理它们,或者只是抛出致命异常并停止处理,直到有人能够弄清楚需要做什么。Protobuf 和 Avro 都有处理未知枚举令牌的优雅方法,如果您的事件格式选择了其中任何一个,则应使用它们。
Enums are often avoided because producers fear creating a new enum token that isn’t present in the consumer’s schema. However, the consumer has a responsibility to consider enum tokens that it does not recognize, and determine if it should process them using a default value or simply throw a fatal exception and halt processing until someone can work out what needs to be done. Both Protobuf and Avro have elegant ways of handling unknown enum tokens and should be used if either is selected for your event format.
一种常见的反模式是type向事件定义添加一个字段,其中不同的type值指示事件的特定子功能。这通常是针对“相似但不同”的数据进行的,并且通常是实施者错误地将事件识别为单一目的的结果。尽管这看起来像是一种节省时间的措施或简化了数据访问模式,但使用type参数重载事件很少是一个好主意。
One common anti-pattern is adding a type field to an event definition, where different type values indicate specific subfeatures of the event. This is generally done for data that is “similar yet different” and is often a result of the implementer incorrectly identifying the events as single-purpose. Though it may seem like a time-saving measure or a simplification of a data access pattern, overloading events with type parameters is rarely a good idea.
这种方法存在几个问题。每个type参数值通常具有根本不同的业务含义,即使其技术表示形式与其他参数值几乎相同。这些含义也可能会随着时间的推移而发生变化,并且事件所涵盖的范围也会逐渐扩大。其中一些类型可能需要添加新参数来跟踪特定于类型的信息,而其他类型则需要单独的参数。最终,您可能会遇到这样的情况:几个非常不同的事件都居住在同一个事件模式中,这使得很难推断事件真正代表什么。
There are several problems with this approach. Each type parameter value usually has a fundamentally different business meaning, even if its technical representation is nearly identical to the others. It is also possible for these meanings to change over time and for the scope that an event covers to creep. Some of these types may require the addition of new parameters to track type-specific information, whereas other types require separate parameters. Eventually you could have a situation where there are several very distinct events all inhabiting the same event schema, making it difficult to reason about what the event truly represents.
这种复杂性不仅影响必须维护和填充这些事件的开发人员,而且还影响数据的消费者,他们需要对发布哪些数据以及发布原因有一致的了解。如果数据合同发生变化,他们希望能够将自己与这些变化隔离开来。添加额外的字段类型要求他们仅过滤他们关心的数据。存在消费者无法完全理解类型的各种含义,从而导致错误消费和逻辑错误代码的风险。对于每个消费者,还必须进行额外的处理以丢弃与该消费者不相关的事件。
This complexity affects not only the developers who must maintain and populate these events, but also the data’s consumers, who need to have a consistent understanding about what data is published and why. If the data contract changes, they expect to be able to isolate themselves from those changes. Adding extra field types requires them to filter for only data that they care about. There is a risk that the consumer will fail to fully understand the various meanings of the types, leading to incorrect consumption and logically wrong code. For each consumer, additional processing must also be done to discard events that aren’t relevant to that consumer.
值得注意的是,添加type字段并不会减少或消除所生成的数据固有的潜在复杂性。事实上,这种复杂性只是从具有不同模式的多个不同事件流转移到合并到一个事件流中的所有模式的联合。可以说这实际上增加了复杂性。模式的未来演变变得更加困难,维护生成事件的代码也变得更加困难。
It is very important to note that adding type fields does not reduce or eliminate the underlying complexity inherent in the data being produced. In fact, this complexity is merely shifted from multiple distinct event streams with distinct schemas to a union of all the schemas merged into one event stream. It could be argued that this actually increases the complexity. Future evolution of the schema becomes more difficult, as does maintaining the code that produces the events.
记住数据契约定义的原则。事件应该与单个业务操作相关,而不是与记录大量数据的通用事件相关。如果您似乎需要具有各种type参数的通用事件,这通常表明您的问题空间和有界上下文没有明确定义。
Remember the principles of the data contract definition. Events should be related to a single business action, not a generic event that records large assortments of data. If it seems like you need a generic event with various type parameters, that’s usually a tell-tale sign that your problem space and bounded context is not well defined.
想象一个简单的网站,用户可以在其中阅读书籍或观看电影。当用户首次访问网站时,例如打开书本或开始观看电影,后端服务会将名为 的此次访问事件发布ProductEngagement到事件流中。这个警示事件的数据结构可能如下所示:
Imagine a simple website where a user can read a book or watch a movie. When the user first engages the website, say by opening the book or starting the movie, a backend service publishes an event of this engagement, named ProductEngagement, into an event stream. The data structure of this cautionary tale event may look something like this:
TypeEnum:书籍、电影
ActionEnum:单击
产品参与度{
产品 ID:长,
产品类型:TypeEnum,
动作类型:ActionEnum
}TypeEnum: Book, Movie
ActionEnum: Click
ProductEngagement {
productId: Long,
productType: TypeEnum,
actionType: ActionEnum
}
现在想象一下新的业务需求出现:您需要在观看电影之前跟踪谁观看了电影预告片。书籍没有预览,尽管布尔值适合观看电影的情况,但它必须可以为空以允许书籍参与。
Now imagine a new business requirement comes in: you need to track who watched the movie trailer before watching the movie. There are no previews for books, and though a boolean would suit the movie-watching case, it must be nullable to allow for book engagements.
产品参与度{
产品 ID:长,
产品类型:TypeEnum,
动作类型:动作枚举,
//仅适用于type=Movie
观看预览:{null,布尔值}
}ProductEngagement {
productId: Long,
productType: TypeEnum,
actionType: ActionEnum,
//Only applies to type=Movie
watchedPreview: {null, Boolean}
}
此时,watchedPreview与 无关Books,但无论如何它都会添加到事件定义中,因为我们已经通过这种方式捕获产品参与度。如果您觉得对下游消费者特别有帮助,您可以在架构中添加注释,告诉他们该字段仅与type=Movie.
At this point, watchedPreview has nothing to do with Books, but it’s added into the event definition anyway since we’re already capturing product engagements this way. If you’re feeling particularly helpful to your downstream consumers, you can add a comment in the schema to tell them that this field is only related to type=Movie.
另一个新的业务需求出现了:您需要跟踪在书中放置书签的用户,并记录该书签所在的页面。同样,由于产品参与只有一个已定义的事件结构,因此您的操作过程仅限于添加新的操作实体 ( Bookmark) 并添加可为空的PageId字段。
Another new business requirement comes in: you need to track users who place a bookmark in their book, and log what page it is on. Again, because there is only a single defined structure of events for product engagements, your course of action is constrained to adding a new action entity (Bookmark) and adding a nullable PageId field.
TypeEnum:书籍、电影
ActionEnum:点击、书签
产品参与度{
产品 ID:长,
产品类型:TypeEnum,
动作类型:动作枚举,
//仅适用于productType=Movie
观看预览:{null,布尔值},
//仅适用于productType=Book,actionType=Bookmark
页面 ID:{null,Int}
}TypeEnum: Book, Movie
ActionEnum: Click, Bookmark
ProductEngagement {
productId: Long,
productType: TypeEnum,
actionType: ActionEnum,
//Only applies to productType=Movie
watchedPreview: {null, Boolean},
//Only applies to productType=Book,actionType=Bookmark
pageId: {null, Int}
}
正如您现在所看到的,业务需求中的一些变化就会使试图服务于多个业务目的的模式变得非常复杂。这增加了数据的生产者和消费者的复杂性,因为他们都必须检查数据逻辑的有效性。要收集和表示的数据总是很复杂,但是通过遵循单一责任原则,您可以将模式分解为更易于管理的内容。让我们看看如果您根据单一职责拆分每个模式,这个示例会是什么样子:
As you can see by now, just a few changes in business requirements can greatly complicate a schema that is trying to serve multiple business purposes. This adds complexity for the producer and the consumer of the data, as they both must check for the validity of the data logic. The data to be collected and represented will always be complex, but by following single responsibility principles you can decompose the schema into something more manageable. Let’s see what this example would look like if you split up each schema according to single responsibilities:
电影点击 {
电影ID:长,
观看预览:布尔值
}
图书点击{
书号:长
}
书签书签 {
书号:长,
页面ID: Int
}MovieClick {
movieId: Long,
watchedPreview: Boolean
}
BookClick {
bookId: Long
}
BookBookmark {
bookId: Long,
pageId: Int
}
和枚举现已消失,并且模式已相应地扁平化productType。actionType现在有三个事件定义,而不仅仅是一个,虽然模式定义数量增加了,但每个模式的内部复杂性都大大降低了。按照每个流一个事件定义的建议,将会看到为每种事件类型创建一个新流。事件定义不会随着时间的推移而发生变化,触发逻辑也不会改变,消费者可以放心单一用途事件定义的稳定性。
The productType and actionType enumerations are now gone, and the schemas have been flattened out accordingly. There are now three event definitions instead of just a single one, and while the schema definition count has increased, the internal complexity of each schema is greatly reduced. Following the recommendation of one event definition per stream would see the creation of a new stream for each event type. Event definitions would not drift over time, the triggering logic would not change, and consumers could be secure in the stability of the single-purpose event definition.
此示例的要点并不是事件定义的原始创建者犯了错误。事实上,当时业务关心任何产品参与度,但不关心具体的产品参与度,所以最初的定义是相当合理的。一旦业务需求更改为包括跟踪特定于电影的事件,服务所有者就需要重新评估事件定义是否仍然服务于单一目的,或者现在是否覆盖多个目的。由于业务需要事件的较低级别详细信息,很明显,虽然该事件可以用于多种目的,但很快就会变得复杂且难以处理。
The takeaway from this example is not that the original creator of the event definition made a mistake. In fact, at the time, the business cared about any product engagements but no specific product engagement, so the original definition is quite reasonable. As soon as the business requirements changed to include tracking movie-specific events, the owner of the service needed to re-evaluate whether the event definition was still serving a single purpose, or if it was instead now covering multiple purposes. Due to the business requiring lower-level details of an event, it became clear that, while the event could serve multiple purposes, it soon would become complex and unwieldy to do so.
避免type在事件中添加超出事件含义的字段。这可能会给事件格式的发展和维护带来很大的困难。
Avoid adding type fields in your events that overload the meaning of the event. This can cause significant difficulty in evolving and maintaining the event format.
花一些时间考虑您的模式可能如何演变。确定所生成数据的主要业务目的、范围、领域以及是否将其构建为单一用途。验证模式是否准确反映业务问题,特别是对于涵盖广泛业务功能职责的系统。可能是业务范围和技术实现不一致。最后,不断变化的业务需求可能需要您重新访问事件定义,并可能将其更改为不仅仅是单个模式的增量定义。如果发生足够的业务变化,事件可能需要完全拆分和重新定义。
Take some time to consider how your schemas may evolve. Identify the main business purpose of the data being produced, the scope, the domain, and whether you’re building it as single-purpose. Validate that the schemas accurately reflect business concerns, especially for systems that cover a broad scope of business function responsibility. It could be that the business scope and the technical implementation are misaligned. Finally, evolving business requirements may require you to revisit the event definitions and potentially change them beyond just incremental definitions of a single schema. Events may need to be split up and redefined completely should sufficient business changes occur.
当事件规模较小、定义明确且易于处理时,其效果会很好。不过,大型活动可能而且确实会发生。一般来说,这些较大的事件代表了大量的上下文信息。也许它们包含与给定事件相关的许多数据点,并且只是对发生的事情的非常大的测量。
Events work well when they’re small, well defined, and easily processed. Large events can and do happen, though. Generally these larger events represent a lot of contextual information. Perhaps they comprise many data points that are related to the given event, and are simply a very large measurement of something that occurred.
当您考虑举办大型活动的设计时,需要考虑几个因素。确保数据与事件直接相关且相关。额外的数据可能已添加到事件“以防万一”,但它可能对下游消费者没有任何实际用途。如果您发现所有事件数据确实直接相关,请退一步看看您的问题空间。您的微服务需要访问数据吗?您可能想要评估有界上下文以查看服务是否正在执行合理的工作量。也许可以通过将附加功能拆分为自己的服务来缩小服务范围。
There are several considerations when you’re looking at a design that produces a very large event. Make sure that the data is directly related and relevant to the event. Additional data may have been added to an event “just in case,” but it may not be of any real use to the downstream consumers. If you find that all the event data is indeed directly related, take a step back and look at your problem space. Does your microservice require access to the data? You may want to evaluate the bounded context to see if the service is performing a reasonable amount of work. Perhaps the service could be reduced in scope with additional functionality split off into its own service.
不过,这种情况并不总是可以避免的,一些事件处理器会生成非常大的输出文件(可能是一个大图像),这些文件太大而无法放入事件流的单个消息中。在这些场景中,您可以使用指向实际数据的指针,但要谨慎使用。这种方法增加了多个事实来源和有效负载可变性的风险,因为不可变的账本无法确保在其系统之外保存数据。
This scenario is not always avoidable, though—some event processors produce very large output files (perhaps a large image) that are much too big to fit into a single message of an event stream. In these scenarios you can use a pointer to the actual data, but do this sparingly. This approach adds risk in the form of multiple sources of truth and payload mutability, as an immutable ledger cannot ensure the preservation of data outside of its system.
在设计新活动时,让该数据的任何预期消费者参与进来非常重要。消费者将比生产者更好地了解自己的需求和预期的业务功能,并可能有助于澄清需求。消费者还将更好地了解他们收到的数据。联合会议或讨论可以解决两个 系统之间数据合同的任何问题。
When designing a new event, it is important to involve any anticipated consumers of this data. Consumers will understand their own needs and anticipated business functions better than the producers and may help in clarifying requirements. Consumers will also get a better understanding of the data coming their way. A joint meeting or discussion can shake out any issues around the data contract between the two systems.
避免使用事件作为信号量或信号。这些事件只是表明发生了某些事情,而不是结果的唯一事实来源。
Avoid using events as a semaphore or a signal. These events simply indicate that something has occurred without being the single source of truth for the results.
考虑一个非常简单的示例,其中系统输出一个事件,指示任意作业的工作已完成。尽管事件本身表明工作已完成,但工作的实际结果并不包含在事件中。这意味着要正确使用此事件,您必须找到已完成的工作实际所在的位置。一旦一段数据有两个真实来源,就会出现一致性问题。
Consider a very simple example where a system outputs an event indicating that work has been completed for an arbitrary job. Although the event itself indicates the work is done, the actual result of the work is not included in the event. This means that to consume this event properly, you must find where the completed work actually resides. Once there are two sources of truth for a piece of data, consistency problems arise.
异步事件驱动架构严重依赖事件质量。高质量事件是使用可演化模式显式定义的,具有明确定义的触发逻辑,并包含带有注释和文档的完整模式定义。隐式模式虽然对于生产者来说更容易实现和维护,但却将大部分解释工作转移给了消费者。由于事件数据丢失和无意的更改,它们也更容易出现意外故障。显式模式是广泛采用事件驱动架构的重要组成部分,特别是当组织不断发展并且无法在组织范围内交流部落知识时。
Asynchronous event-driven architectures rely heavily upon event quality. High-quality events are explicitly defined with an evolvable schema, have well-defined triggering logic, and include full schema definitions with comments and documentation. Implicit schemas, while easier to implement and maintain for the producer, offload much of the interpretation work onto the consumer. They are also more prone to unexpected failures due to missing event data and unintentional changes. Explicit schemas are an essential component for widespread adoption of event-driven architectures, particularly as an organization grows and it becomes impossible to communicate tribal knowledge organization-wide.
事件定义应该狭窄并密切关注事件的领域。事件应该代表特定的业务事件并包含适当的数据字段来记录所发生的情况。这些事件形成了有关业务运营的官方叙述,并且可以由其他微服务根据自己的需求来使用。
Event definitions should be narrow and closely focused on the domain of the event. An event should represent a specific business occurrence and contain the appropriate data fields to record what has happened. These events form the official narrative about business operations and can be consumed by other microservices for their own needs.
模式演化是显式模式的一个非常重要的方面,因为它允许事件域模型的变化的受控机制。领域模型的发展是很常见的,特别是随着新业务需求的出现和组织的扩展。模式演化允许生产者和消费者将自己与对其操作并不重要的更改隔离开来,同时允许那些关心更改的服务相应地进行自我更新。
Schema evolution is a very important aspect of explicit schemas, as it allows for a controlled mechanism of change for the event domain model. It is common for a domain model to evolve, particularly as new business requirements emerge and the organization expands. Schema evolution allows producers and consumers to isolate themselves from changes that aren’t essential to their operations, while permitting those services that do care about the changes to update themselves accordingly.
在某些情况下,模式演化是不可能的,必须发生重大变化。生产者和消费者利益相关者必须沟通重大变更背后的原因,并共同重新定义未来的领域模型。旧事件的迁移可能有必要,也可能没有必要。
In some cases schema evolution is not possible, and a breaking change must occur. The producer and consumer stakeholders must communicate the reasons behind the breaking changes and come together to redefine the domain model going forward. Migration of old events may or may not be necessary.
将组织转变为事件驱动架构需要将现有系统集成到生态系统中。您的组织可能拥有一个或多个整体关系数据库应用程序。各种实现之间可能存在点对点连接。也许已经存在用于在系统之间传输批量数据的类似事件的机制,例如通过中间文件存储位置定期同步数据库转储。如果您正在从头开始构建事件驱动的微服务架构并且没有遗留系统,那就太棒了!您可以跳过本节(尽管您可能应该考虑 EDM 可能不适合您的新项目)。但是,如果您有需要支持的现有旧系统,请继续阅读。
Transitioning an organization to an event-driven architecture requires the integration of existing systems into the ecosystem. Your organization may have one or more monolithic relational database applications. Point-to-point connections between various implementations are likely to exist. Perhaps there are already event-like mechanisms for transferring bulk data between systems, such as regular syncing of database dumps via an intermediary file store location. In the case that you are building an event-driven microservice architecture from the ground up and have no legacy systems, great! You can skip this section (though perhaps you should consider that EDM may not be right for your new project). However, if you have existing legacy systems that need to be supported, read on.
在任何业务领域中,都存在跨多个子域通常需要的实体和事件。例如,电子商务零售商需要向各种有界上下文提供产品信息、价格、库存和图像。也许付款是由一个系统收集的,但需要在另一个系统中进行验证,并在第三个系统中对购买模式进行分析。将这些数据作为新的单一事实来源在中央位置提供,允许每个系统在数据可用时使用它。迁移到事件驱动的微服务需要在事件代理中提供必要的业务域数据,并可作为事件流使用。这样做的过程称为数据解放,并且涉及数据源。来自现有系统和包含它的状态存储的数据。
In any business domain, there are entities and events that are commonly required across multiple subdomains. For example, an ecommerce retailer will need to supply product information, prices, stock, and images to various bounded contexts. Perhaps payments are collected by one system but need to be validated in another, with analytics on purchase patterns performed in a third system. Making this data available in a central location as the new single source of truth allows each system to consume it as it becomes available. Migrating to event-driven microservices requires making the necessary business domain data available in the event broker, consumable as event streams. Doing so is a process known as data liberation, and involves sourcing the data from the existing systems and state stores that contain it.
事件流中生成的数据可以由任何系统(事件驱动或其他系统)访问。虽然事件驱动的应用程序可以使用流框架和本机使用者来读取事件,但由于技术和性能限制等多种因素,遗留应用程序可能无法轻松访问它们。在这种情况下,您可能需要将事件从事件流接收到现有的状态存储中。
Data produced to an event stream can be accessed by any system, event-driven or otherwise. While event-driven applications can use streaming frameworks and native consumers to read the events, legacy applications may not be able to access them as easily due to a variety of factors, such as technology and performance limitations. In this case, you may need to sink the events from an event stream into an existing state store.
有许多用于获取和接收事件数据的模式和框架。对于每种技术,本章将介绍其必要性、如何实施以及与不同方法相关的权衡。然后,我们将回顾数据解放和数据下沉如何适应整个组织、它们所产生的影响以及构建成功努力的方法。
There are a number of patterns and frameworks for sourcing and sinking event data. For each technique, this chapter will cover why it’s necessary, how to do it, and the tradeoffs associated with different approaches. Then, we’ll review how data liberation and sinking fit in to the organization as a whole, the impacts they have, and ways to structure your efforts for success.
数据解放是跨域数据集的识别和发布到其相应的事件流,是事件驱动架构迁移策略的一部分。跨域数据集包括存储在一个数据存储中且其他外部系统所需的任何数据。现有服务和数据存储之间的点对点依赖关系往往突出了应该释放的跨域数据,如图4-1所示,其中三个依赖服务直接查询遗留系统。
Data liberation is the identification and publication of cross-domain data sets to their corresponding event streams and is part of a migration strategy for event-driven architectures. Cross-domain data sets include any data stored in one data store that is required by other external systems. Point-to-point dependencies between existing services, and data stores often highlight the cross-domain data that should be liberated, as shown in Figure 4-1, where three dependent services are querying the legacy system directly.
数据解放强化了事件驱动架构的两个主要特征:单一事实来源和消除系统之间的直接耦合。解放的事件流允许将新的事件驱动的微服务构建为消费者,并在适当的时候迁移现有系统。现在可以使用反应式事件驱动框架和服务来消费和处理数据,下游消费者不再需要直接耦合到源数据系统。
Data liberation enforces two primary features of event-driven architecture: the single source of truth and the elimination of direct coupling between systems. The liberated event streams allow new event-driven microservices to be built as consumers, with existing systems migrated in due time. Reactive event-driven frameworks and services may now be used to consume and process data, and downstream consumers are no longer required to directly couple on the source data system.
通过充当单一事实来源,这些流还标准化了整个组织的系统访问数据的方式。系统不再需要直接耦合到底层数据存储和应用程序,而是可以仅耦合到事件流的数据契约。解放后的工作流程如图4-2所示。
By serving as a single source of truth, these streams also standardize the way in which systems across the organization access data. Systems no longer need to couple directly to the underlying data stores and applications, but instead can couple solely on the data contracts of event streams. The post-liberation workflow is shown in Figure 4-2.
数据集及其释放的事件流必须保持完全同步,尽管由于事件传播的延迟,此要求仅限于最终一致性。释放的事件流必须具体化回源表的精确副本,并且此属性广泛用于事件驱动的微服务(如第 7 章所述)。相比之下,遗留系统不会从任何事件流重建其数据集,而是通常具有自己的备份和恢复机制,并且从释放的事件流中绝对不读回任何内容。
A data set and its liberated event stream must be kept fully in sync, although this requirement is limited to eventual consistency due to the latency of event propagation. A stream of liberated events must materialize back into an exact replica of the source table, and this property is used extensively for event-driven microservices (as covered in Chapter 7). In contrast, legacy systems do not rebuild their data sets from any event streams, but instead typically have their own backup and restore mechanisms and read absolutely nothing back from the liberated event stream.
在完美的世界中,所有状态都将从事件流的单一事实源创建、管理、维护和恢复。任何共享状态都应首先发布到事件代理,然后具体化回任何需要具体化状态的服务,包括首先生成数据的服务,如图4-3所示。
In the perfect world, all state would be created, managed, maintained, and restored from the single source of truth of the event streams. Any shared state should be published to the event broker first and materialized back to any services that need to materialize the state, including the service that produced the data in the first place, as shown in Figure 4-3.
虽然在事件代理中维护状态的理想方案对于新的微服务和重构的遗留应用程序来说是可以实现的,但它不一定对所有应用程序都可用或实用。对于除了与变更数据捕获机制的初始集成之外不太可能被重构或更改的服务尤其如此。遗留系统对于组织来说既极其重要,又难以重构,最严重的违规者被认为是一个大泥球。尽管系统很复杂,但其内部数据仍然需要由其他新系统访问。虽然重构可能绝对是可取的,但在现实中存在许多问题阻止这种情况发生:
While the ideal of maintaining state in the event broker is accessible for new microservices and refactored legacy applications, it is not necessarily available or practical for all applications. This is particularly true for services that are unlikely to ever be refactored or changed beyond initial integration with change-data capture mechanisms. Legacy systems can be both extremely important to the organization and prohibitively difficult to refactor, with the worst offenders being considered a big ball of mud. Despite the complexity of a system, their internal data will still need to be accessed by other new systems. While refactoring may absolutely be desirable, there are a number of issues that prevent this from happening in reality:
许多遗留系统只有最少的开发人员支持,并且需要省力的解决方案来生成解放的数据。
Many legacy systems have minimal developer support and require low-effort solutions to generate liberated data.
将现有的应用程序工作流程重新设计为异步事件驱动和同步 MVC(模型-视图-控制器)Web 应用程序逻辑的混合体可能会非常昂贵,尤其是对于复杂的遗留单体而言。
Reworking the preexisting application workflows into a mix of asynchronous event-driven and synchronous MVC (Model-View-Controller) web application logic may be prohibitively expensive, especially for complex legacy monoliths.
对遗留系统进行的更改可能会产生意想不到的后果,特别是当系统的职责由于技术债务和与其他系统的未识别的点对点连接而不清楚时。
Changes made to legacy systems may have unintended consequences, especially when the system’s responsibilities are unclear due to technical debt and unidentified point-to-point connections with other systems.
这里有一个妥协的机会。您可以使用数据解放模式从数据存储中提取数据并创建必要的事件流。这是单向事件驱动架构的一种形式,因为遗留系统不会从释放的事件流中读回,如图4-3所示。相反,根本目标是通过严格控制事件数据的发布来保持内部数据集与外部事件流同步。事件流最终将与遗留应用程序的内部数据集保持一致,如图4-4所示。
There is an opportunity for compromise here. You can use data liberation patterns to extract the data out of the data store and create the necessary event streams. This is a form of unidirectional event-driven architecture, as the legacy system will not be reading back from the liberated event stream, as shown in Figure 4-3. Instead, the fundamental goal is to keep the internal data set synchronized with the external event stream through strictly controlled publishing of event data. The event stream will be eventually consistent with the internal data set of the legacy application, as shown in Figure 4-4.
与任何其他事件一样,解放的数据也遵循第 3 章中介绍的相同的图式化建议。定义良好的事件流的属性之一是它包含的事件有一个明确定义的且进化兼容的模式。您应该确保消费者拥有基本的数据质量保证,作为架构定义的数据契约的一部分。模式的改变只能根据进化规则进行。
Liberated data, much like any other event, is subject to the same recommendations of schematization that were introduced in Chapter 3. One of the properties of a well-defined event stream is that there is an explicitly defined and evolutionarily compatible schema for the events it contains. You should ensure that consumers have basic data quality guarantees as part of the data contract defined by the schema. Changes to the schema can only be made according to evolutionary rules.
对整个组织中的自由事件数据和本机事件数据使用相同的标准格式。
Use the same standard format for both liberated event data and native event data across your organization.
根据定义,在整个业务中最相关和使用的数据是最需要解放的数据。对源的数据定义进行的更改(例如创建新字段、更改现有字段或删除其他字段)可能会导致动态更改的数据向下游传播到消费者。未能对解放的数据使用显式定义的模式将迫使下游消费者解决任何不兼容性。这对于提供单一事实来源来说是非常有问题的,因为下游消费者不应该尝试自己解析或解释数据。提供所生成数据的可靠且最新的模式并仔细考虑数据随时间的演变非常重要。
By definition, the data that is most relevant and used across the business is the data that is most necessary to liberate. Changes made to the data definitions of the source, such as creating new fields, altering existing ones, or dropping others, can result in dynamically changing data being propagated downstream to consumers. Failing to use an explicitly defined schema for liberated data will force downstream consumers to resolve any incompatibilities. This is extremely problematic for the provision of the single source of truth, as downstream consumers should not be attempting to parse or interpret data on their own. It is extremely important to provide a reliable and up-to-date schema of the produced data and to carefully consider the evolution of the data over time.
您可以使用三种主要的数据解放模式从底层数据存储中提取数据。由于解放的数据旨在形成新的单一事实来源,因此它必须包含来自数据存储的整个数据集。此外,这些数据必须随着新的插入、更新和删除而保持最新。
There are three main data liberation patterns that you can use to extract data from the underlying data store. Since liberated data is meant to form the new single source of truth, it follows that it must contain the entire set of data from the data store. Additionally, this data must be kept up to date with new insertions, updates, and deletes.
您可以通过查询底层状态存储来提取数据。这可以在任何数据存储上执行。
You extract data by querying the underlying state store. This can be performed on any data store.
您可以通过遵循仅附加日志来提取数据以了解底层数据结构的更改。此选项仅适用于维护数据修改日志的选定数据存储。
You extract data by following the append-only log for changes to the underlying data structures. This option is available only for select data stores that maintain a log of the modifications made to the data.
在此模式中,您首先将数据推送到用作输出队列的表。另一个线程或单独的进程查询该表,将数据发送到相关事件流,然后删除关联的条目。此方法要求数据存储支持事务和输出队列机制,通常是配置为用作队列的独立表。
In this pattern, you first push data to a table used as an output queue. Another thread or separate process queries the table, emits the data to the relevant event stream, and then deletes the associated entries. This method requires that the data store support both transactions and an output queue mechanism, usually a standalone table configured for use as a queue.
虽然每种模式都是独一无二的,但三者之间有一个共同点。updated_at每个事件都应按排序的时间戳顺序生成其事件,并在其输出事件记录标头中使用源记录的最新时间。这将生成一个事件流,该事件流的时间戳是根据事件的发生时间而不是生产者发布事件的时间。这对于数据解放尤其重要,因为它准确地表示了工作流中事件实际发生的时间。基于时间戳的事件交错将在第 6 章中进一步讨论。
While each pattern is unique, there is one commonality among the three. Each should produce its events in sorted timestamp order, using the source record’s most recent updated_at time in its output event record header. This will generate an event stream timestamped according to the event’s occurrence, not the time that the producer published the event. This is particularly important for data liberation, as it accurately represents when events actually happened in the workflow. Timestamp-based interleaving of events is discussed further in Chapter 6.
释放数据的一种方法涉及使用专用的集中式框架将数据提取到事件流中。用于捕获事件流的集中式框架的示例包括Kafka Connect(专门用于 Kafka 平台)、Apache Gobblin和Apache NiFi。每个框架都允许您对基础数据集执行查询,并将结果通过管道传输到输出事件流。每个选项也是可扩展的,以便您可以添加更多实例以提高执行变更数据捕获 (CDC) 作业的能力。它们支持与Confluence (Apache Kafka) 提供的模式注册表的各种级别的集成,但当然可以执行自定义以支持其他模式注册表。有关详细信息,请参阅“架构注册表” 。
One method of liberating data involves the usage of a dedicated, centralized framework to extract data into event streams. Examples of centralized frameworks for capturing event streams include Kafka Connect (exclusively for the Kafka platform), Apache Gobblin, and Apache NiFi. Each framework allows you to execute a query against the underlying data set with the results piped through to your output event streams. Each option is also scalable, such that you can add further instances to increase the capacity for executing change-data capture (CDC) jobs. They support various levels of integration with the schema registry offered by Confluent (Apache Kafka), but customization can certainly be performed to support other schema registries. See “Schema Registry” for more information.
并非所有数据解放过程都需要专用框架,并且许多系统更适合直接拥有自己的事件流数据生产。事实上,这些框架无意中鼓励了数据访问反模式。最常见的反模式之一是将内部数据模型暴露给外部系统,进一步增加而不是减少耦合,这是事件驱动架构的主要好处之一。这将在本章的其余部分进一步讨论。
Not all data liberation processes require a dedicated framework, and many systems are better suited to taking direct ownership of their own event stream data production. In fact, these frameworks inadvertently encourage data access anti-patterns. One of the most common anti-patterns is the exposure of internal data models to external systems, further increasing coupling instead of decreasing it, as is one of the major benefits of event-driven architectures. This will be covered further in the remainder of the chapter.
基于查询的数据解放涉及查询数据存储并将选定的结果发送到关联的事件流。客户端用于使用适当的 API、SQL 或类似 SQL 的语言从数据存储请求特定数据集。必须批量查询数据集以提供事件的历史记录。然后定期更新,确保输出事件流发生变化。
Query-based data liberation involves querying the data store and emitting selected results to an associated event stream. A client is used to request the specific data set from the data store using the appropriate API, SQL, or SQL-like language. A data set must be bulk-queried to provide the history of events. Periodic updates then follow, ensuring that changes are produced to the output event stream.
此模式中使用了多种类型的查询。
There are several types of queries used in this pattern.
批量加载查询并加载数据集中的所有数据。当需要在每个轮询间隔加载整个表时以及在正在进行的增量更新之前,执行批量加载。
Bulk loading queries and loads all of the data from the data set. Bulks loads are performed when the entire table needs to be loaded at each polling interval, as well as prior to ongoing incremental updates.
批量加载可能会很昂贵,因为它需要从数据存储中获取整个数据集。对于较小的数据集,这往往不是问题,但大型数据集,尤其是具有数百万或数十亿条记录的数据集,可能很难获取。对于查询和处理非常大的数据集,我建议您研究特定数据存储的最佳实践,因为这些实践可能会因实施而有很大差异。
Bulk loading can be expensive, as it requires obtaining the entire data set from the data store. For smaller data sets this tends not to be a problem, but large data sets, especially those with millions or billions of records, may be difficult to obtain. For querying and processing very large data sets I recommend you research best practices for your particular data store, since these can vary significantly with implementation.
通过增量时间戳加载,您可以查询并加载自上一个查询结果的最高时间戳以来的所有数据。此方法使用updated-at数据集中的列或字段来跟踪记录上次修改的时间。在每次增量更新期间,仅updated-at查询时间戳晚于上次处理时间的记录。
With incremental timestamp loading, you query and load all data since the highest timestamp of the previous query’s results. This approach uses an updated-at column or field in the data set that keeps track of the time when the record was last modified. During each incremental update, only records with updated-at timestamps later than the last processed time are queried.
自增ID加载涉及查询并加载所有大于ID最后值的数据。这需要严格排序的自动增量Integer或Long字段。每次增量更新时,只查询ID大于上次处理的ID的记录。这种方法通常用于查询具有不可变记录的表,例如使用发件箱表时(请参阅“使用更改数据捕获日志解放数据”)。
Autoincrementing ID loading involves querying and loading all data larger than the last value of the ID. This requires a strictly ordered autoincrementing Integer or Long field. During each incremental update, only records with an ID larger than the last processed ID are queried. This approach is often used for querying tables with immutable records, such as when using the outbox tables (see “Liberating Data Using Change-Data Capture Logs”).
自定义查询仅受客户端查询语言的限制。当客户端仅需要较大数据集中的某个数据子集时,或者在连接和非规范化多个表中的数据以避免内部数据模型过度暴露时,通常会使用此方法。例如,用户可以根据特定字段过滤业务合作伙伴数据,其中每个合作伙伴的数据被发送到其自己的事件流。
A custom query is limited only by the client querying language. This approach is often used when the client requires only a certain subset of data from a larger data set, or when joining and denormalizing data from multiple tables to avoid over-exposure of the internal data model. For instance, a user could filter business partner data according to a specific field, where each partner’s data is sent to its own event stream.
任何增量更新的第一步是确保数据集记录中提供必要的时间戳或自动增量 ID。必须有一个字段,查询可以使用该字段从尚未处理的记录中筛选出已处理的记录。缺少这些字段的数据集需要添加它们,并且需要配置数据存储以填充必要的updated_at
时间戳或自动增量 ID 字段。如果无法将字段添加到数据集中,则基于查询的模式将无法进行增量更新。
The first step of any incremental update is to ensure that the necessary timestamp or autoincrementing ID is available in the records of your data set. There must be a field that the query can use to filter out records it has already processed from those it has yet to process. Data sets that lack these fields will need to have them added, and the data store will need to be configured to populate the necessary updated_at
timestamp or the autoincrementing ID field. If the fields cannot be added to the data set, then incremental updates will not be possible with a query-based pattern.
第二步是确定轮询频率和更新延迟。更高频率的更新为下游数据更新提供更低的延迟,尽管这是以数据存储上更大的总负载为代价的。考虑请求之间的间隔是否足以完成所有数据的加载也很重要。在旧查询仍在加载时开始新查询可能会导致竞争条件,即旧数据会覆盖输出事件流中的新数据。
The second step is to determine the frequency of polling and the latency of the updates. Higher-frequency updates provide lower latency for data updates downstream, though this comes at the expense of a larger total load on the data store. It’s also important to consider whether the interval between requests is sufficient to finish loading all of the data. Beginning a new query while the old one is still loading can lead to race conditions, where older data overwrites newer data in the output event streams.
选择增量更新字段并确定更新频率后,最后一步是在启用增量更新之前执行单个批量加载。在进一步增量更新之前,此批量加载必须查询并生成数据集中的所有现有数据。
Once the incremental update field has been selected and the frequency of updates determined, the final step is to perform a single bulk load before enabling incremental updates. This bulk load must query and produce all of the existing data in the data set prior to further incremental updates.
Query-based updating has a number of advantages, including:
可以查询任何数据存储,并且可以使用所有用于查询的客户端选项。
Any data store can be queried, and the entire range of client options for querying is available.
特定查询可以更频繁地执行以满足更严格的 SLA(服务级别协议),而其他更昂贵的查询可以不那么频繁地执行以节省资源。
Specific queries can be executed more frequently to meet tighter SLAs (service-level agreements), while other more expensive queries can be executed less frequently to save resources.
关系数据库可以通过使用底层数据的视图或物化视图来提供与内部数据模型的隔离。此技术可用于隐藏不应在数据存储外部公开的域模型信息。
Relational databases can provide isolation from the internal data model by using views or materialized views of the underlying data. This technique can be used to hide domain model information that should not be exposed outside of the data store.
请记住,释放的数据将是唯一的事实来源。考虑是否应该释放任何隐藏或遗漏的数据,或者是否需要重构源数据模型。这种情况通常发生在从遗留系统中解放数据的过程中,其中业务数据和实体数据随着时间的推移变得交织在一起。
Remember that the liberated data will be the single source of truth. Consider whether any concealed or omitted data should instead be liberated, or if the source data model needs to be refactored. This often occurs during data liberation from legacy systems, where business data and entity data have become intertwined over time.
There are some downsides to query-based updating as well:
updated-at时间戳updated-at timestamp要查询的事件的基础表或命名空间必须有一列包含其updated-at时间戳。这对于跟踪数据的上次更新时间和进行增量更新至关重要。
The underlying table or namespace of events to query must have a column containing their updated-at timestamp. This is essential for tracking the last update time of the data and for making incremental updates.
硬删除不会显示在查询结果中,因此跟踪删除仅限于基于标志的软删除,例如布尔is_deleted列。
Hard deletions will not show up in the query results, so tracking deletions is limited to flag-based soft deletions, such as a boolean is_deleted column.
可能会发生与下游事件格式架构规则不兼容的数据集架构更改。如果解放机制与数据存储应用程序的代码库分离(基于查询的系统通常就是这种情况),那么破坏的可能性就会越来越大。
Data set schema changes may occur that are incompatible with downstream event format schema rules. Breakages are increasingly likely if the liberation mechanism is separate from the code base of the data store application, which is usually the case for query-based systems.
数据仅在轮询间隔同步,因此对同一记录的一系列单独更改可能仅显示为单个事件。
Data is synced only at polling intervals, and so a series of individual changes to the same record may only show up as a single event.
查询使用底层系统资源来执行,这可能会导致生产系统出现不可接受的延迟。可以通过使用只读副本来缓解此问题,但会产生额外的财务成本和系统复杂性。
Queries use the underlying system resources to execute, which can cause unacceptable delays on a production system. This issue can be mitigated by the use of a read-only replica, but additional financial costs and system complexity will apply.
查询和返回的数据量根据底层数据的变化而变化。在最坏的情况下,每次都会更改整个数据体。当一个查询在下一个查询开始之前尚未完成时,这可能会导致竞争条件。
The quantity of data queried and returned varies depending on changes made to the underlying data. In the worst-case scenario, the entire body of data is changed each time. This can result in race conditions when a query is not finished before the next one starts.
解放数据的另一种模式是使用数据存储的底层更改数据捕获日志( MySQL 中的二进制日志, PostgreSQL 中的预写日志)作为信息源。这是一种仅附加的数据记录结构,详细说明了跟踪数据集随时间推移发生的所有情况。这些更改包括创建、删除和更新单个记录,以及创建、删除和更改单个数据集及其架构。
Another pattern for liberating data is using the data store’s underlying change-data capture logs (binary logs in MySQL, write-ahead logs for PostgreSQL) as the source of information. This is an append-only data logging structure that details everything that has happened to the tracked data sets over time. These changes include the creation, deletion, and updating of individual records, as well as the creation, deletion, and altering of the individual data sets and their schemas.
变更数据捕获的技术选项比基于查询的捕获的技术选项要窄。并非所有数据存储都实现了不可变的更改记录,并且在那些实现了的数据存储中,并非所有数据存储都具有可用于提取数据的现成连接器。这种方法主要适用于选定的关系数据库,例如 MySQL 和 PostgreSQL,尽管任何具有一组全面变更日志的数据存储都是合适的候选者。许多其他现代数据存储公开事件 API,充当物理预写日志的代理。例如,MongoDB 提供Change Streams接口,而 Couchbase 通过其内部复制协议提供复制访问。
The technology options for change-data capture are narrower than those for query-based capturing. Not all data stores implement an immutable logging of changes, and of those that do, not all of them have off-the-shelf connectors available for extracting the data. This approach is mostly applicable to select relational databases, such as MySQL and PostgreSQL, though any data store with a set of comprehensive changelogs is a suitable candidate. Many other modern data stores expose event APIs that act as a proxy for a physical write-ahead log. For example, MongoDB provides a Change Streams interface, whereas Couchbase provides replication access via its internal replication protocol.
数据存储日志不太可能包含自开始以来的所有更改,因为它可能是大量数据并且通常不需要保留。在从数据存储日志开始更改数据捕获过程之前,您需要拍摄现有数据的快照。此快照通常涉及对表进行大型、影响性能的查询,通常称为引导。您必须确保引导查询结果中的记录与日志中的记录之间存在重叠,这样您就不会意外错过任何记录。
The data store log is unlikely to contain all changes since the beginning of time, as it can be a huge amount of data and is usually not necessary to retain. You will need to take a snapshot of the existing data prior to starting the change-data capture process from the data store’s log. This snapshot usually involves a large, performance-impacting query on the table and is commonly referred to as bootstrapping. You must ensure that there is overlap between the records in the bootstrapped query results and the records in the log, such that you do not accidentally miss any records.
从变更日志中捕获事件时,您必须检查点进度,但根据您使用的工具,这可能已经内置。如果变更数据捕获机制失败,检查点将用于恢复最后存储的变更日志索引。这种方法只能提供至少一次的记录生成,这往往适合数据解放的基于实体的性质。由于更新实体数据是幂等的,因此生成附加记录是无关紧要 的。
You must checkpoint progress when capturing events from the changelogs, though depending on the tooling you use, this may already be built in. In the event that the change-data capture mechanism fails, the checkpoint is used to restore the last stored changelog index. This approach can only provide at-least-once production of records, which tends to be suitable for the entity-based nature of data liberation. The production of an additional record is inconsequential since updating entity data is idempotent.
有许多选项可用于从变更日志中获取数据。Debezium是关系数据库最受欢迎的选择之一,因为它支持最常见的数据库。Debezium 可以为两者生成记录Apache Kafka 和 Apache Pulsar 及其现有实现。对更多经纪商的支持当然是可能的,尽管它可能需要一些内部开发工作。Maxwell是二进制日志读取器选项的另一个示例,尽管它目前仅限于支持 MySQL 数据库,并且只能向 Apache Kafka 生成数据。
There are a number of options available for sourcing data from changelogs. Debezium is one of the most popular choices for relational databases, as it supports the most common ones. Debezium can produce records to both Apache Kafka and Apache Pulsar with its existing implementations. Support for additional brokers is certainly possible, though it may require some in-house development work. Maxwell is another example of a binary log reader option, though it is currently limited in support to just MySQL databases and can produce data only to Apache Kafka.
图 4-5显示了发出二进制变更日志的 MySQL 数据库。运行 Debezium 连接器的 Kafka Connect 服务正在使用原始二进制日志。Debezium 解析数据并将其转换为离散事件。接下来,事件路由器根据该事件的源表将每个事件发送到 Kafka 中的特定事件流。下游消费者现在可以通过使用来自 Kafka 的相关事件流来访问数据库内容。
Figure 4-5 shows a MySQL database emitting its binary changelog. A Kafka Connect service, running a Debezium connector, is consuming the raw binary log. Debezium parses the data and converts it into discrete events. Next, an event router emits each event to a specific event stream in Kafka, depending on the source table of that event. Downstream consumers are now able to access the database content by consuming the relevant event streams from Kafka.
Some benefits of using data store logs include:
Binary logs contain hard record deletions. These can be converted into delete events without the need for soft deletes as in query-based updates.
对于使用预写和二进制日志的数据存储,可以执行更改数据捕获,而不会对数据存储的性能产生任何影响。对于那些使用更改表的系统(例如在 SQL Server 中),影响与数据量有关。
For data stores that use write-ahead and binary logs, change-data capture can be performed without any impact to the data store’s performance. For those that use change tables, such as in SQL Server, the impact is related to the volume of data.
一旦事件写入二进制和预写日志,就可以传播更新。与其他数据释放模式相比,这会导致非常低的延迟。
Updates can be propagated as soon as the event is written to the binary and write-ahead logs. This results in very low latency when compared to other data liberation patterns.
以下是使用数据库日志的一些缺点:
The following are some of the downsides to using data base logs:
内部数据模型在变更日志中完全公开。与基于查询的更新不同,必须仔细且有选择地管理底层数据模型的隔离,其中视图可用于提供隔离。
The internal data model is completely exposed in the changelogs. Isolation of the underlying data model must be carefully and selectively managed, unlike query-based updating, where views can be used to provide isolation.
变更日志仅包含事件数据。一些 CDC 机制可以从物化视图中提取,但对于许多其他机制,非规范化必须在数据存储之外进行。这可能会导致创建高度规范化的事件流,从而需要下游微服务处理外键连接和反规范化。
Changelogs contain only the event data. Some CDC mechanisms can extract from materialized views, but for many others, denormalization must occur outside of the data store. This may lead to the creation of highly normalized event streams, requiring downstream microservices to handle foreign-key joins and denormalization.
与基于查询的数据解放过程非常相似,基于二进制日志的过程存在于数据存储应用程序之外。有效的数据存储更改(例如更改数据集或重新定义字段类型)可能与事件模式的特定演化规则完全不兼容。
Much like the query-based data liberation process, the binary-log-based process exists outside of the data store application. Valid data store changes, such as altering a data set or redefining a field type, may be completely incompatible for the specific evolution rules of the event schema.
发件箱表包含对数据存储的内部数据所做的显着更改,每个重要更新都存储为其自己的行。每当对标记为更改数据捕获的数据存储表之一进行插入、更新或删除时,就可以将相应的记录发布到发件箱表。更改数据捕获下的每个表都可以有自己的发件箱表,或者可以将单个发件箱用于所有更改(稍后将详细介绍)。
An outbox table contains notable changes made to the internal data of a data store, with each significant update stored as its own row. Whenever an insert, update, or delete is made to one of the data store tables marked for change-data capture, a corresponding record can be published to the outbox table. Each table under change-data capture can have its own outbox table, or a single outbox can be used for all changes (more on this shortly).
内表更新和发件箱更新都必须捆绑到单个事务中,以便只有在整个事务成功时才会发生每个更新。如果不这样做,最终可能会导致与作为单一事实来源的事件流发生分歧,这可能难以检测和修复。这种模式是一种更具侵入性的变更数据捕获方法,因为它需要修改数据存储或应用程序层,而这两者都需要数据存储开发人员的参与。发件箱表模式利用数据存储的持久性为等待发布到外部事件流的事件提供预写日志。
Both the internal table updates and the outbox updates must be bundled into a single transaction, such that each occurs only if the entire transaction succeeds. A failure to do may eventually result in divergence with the event stream as the single source of truth, which can be difficult to detect and repair. This pattern is a more invasive approach to change-data capture as it requires modification to either the data store or the application layer, both of which require the involvement of the data store developers. The outbox table pattern leverages the durability of the data store to provide a write-ahead log for events awaiting to be published to external event streams.
发件箱表中的记录必须具有严格的排序标识符,因为同一主键可能会在短时间内更新多次。或者,您可以覆盖该主键的先前更新,但这需要首先找到先前的条目并引入额外的性能开销。这也意味着被覆盖的记录不会被发送到下游。
The records in outbox tables must have a strict ordering identifier, for the same primary key may be updated many times in short order. Alternatively, you could overwrite the previous update for that primary key, though this requires finding the previous entry first and introduces additional performance overhead. It also means that the overwritten record will not be emitted downstream.
在插入时分配的自动递增 ID 最适合用来确定事件的发布顺序。还应该维护时间戳created_at列,因为它反映了在数据存储中创建记录的事件时间,并且可以在发布到事件流期间使用它来代替挂钟时间。这将允许事件调度程序进行准确的交错,如第 6 章所述。
An autoincrementing ID, assigned at insertion time, is best used to determine the order in which the events are to be published. A created_at timestamp column should also be maintained, as it reflects the event time that the record was created in the data store and can be used instead of the wall-clock time during publishing to the event stream. This will allow accurate interleaving by the event scheduler as discussed in Chapter 6.
图4-6显示了端到端的工作流程。数据存储客户端对内部表进行的更新包含在一个事务中,并更新了发件箱表,这样任何失败都可以确保两者之间的数据保持一致。同时,使用单独的应用程序线程或进程来不断轮询发件箱并将数据生成到相应的事件流。一旦成功生成,发件箱中相应的记录将被删除。如果出现任何故障,无论是数据存储、消费者/生产者还是事件代理本身,发件箱记录仍将保留,不会有丢失的风险。此模式提供至少一次交付 保证。
Figure 4-6 shows the end-to-end workflow. Updates to internal tables made by the data store client are wrapped in a transaction with an update to the outbox table, such that any failures ensures data remains consistent between the two. Meanwhile, a separate application thread or process is used to continually poll the outboxes and produce the data to the corresponding event streams. Once successfully produced, the corresponding records in the outbox are deleted. In the case of any failure, be it the data store, the consumer/producer, or the event broker itself, outbox records will still be retained without risk of loss. This pattern provides at-least-once delivery guarantees.
包含发件箱表会给数据存储及其请求处理应用程序带来额外的负载。对于负载最小的小型数据存储,开销可能完全不会被注意到。或者,对于非常大的数据存储,尤其是那些具有大量负载和捕获的许多表的数据存储,它可能非常昂贵。应根据具体情况评估此方法的成本,并与响应策略(例如解析变更数据捕获日志)的成本进行平衡。
The inclusion of outbox tables introduces additional load on the data store and its request-handling applications. For small data stores with minimal load, the overhead may go completely unnoticed. Alternately, it may be quite expensive with very large data stores, particularly those with significant load and many tables under capture. The cost of this approach should be evaluated on a case-by-case basis and balanced against the costs of a reactive strategy such as parsing the change-data capture logs.
发件箱不需要与内部表进行 1:1 映射。事实上,发件箱的主要好处之一是数据存储客户端可以将内部数据模型与下游消费者隔离。域的内部数据模型可能使用许多高度规范化的表,这些表针对关系操作进行了优化,但很大程度上不适合下游消费者的消费。即使简单的域也可能包含多个表,如果将其作为独立的流公开,则需要重建以供下游消费者使用。就操作开销而言,这很快就会变得极其昂贵,因为多个下游团队将不得不重建域模型并处理事件流中的关系数据。
An outbox does not need to map 1:1 with an internal table. In fact, one of the major benefits of the outbox is that the data store client can isolate the internal data model from downstream consumers. The internal data model of the domain may use a number of highly normalized tables that are optimized for relational operations but are largely unsuitable for consumption by downstream consumers. Even simple domains may comprise multiple tables, which if exposed as independent streams, would require reconstruction for usage by downstream consumers. This quickly becomes extremely expensive in terms of operational overhead, as multiple downstream teams will have to reconstruct the domain model and deal with handling relational data in event streams.
将内部数据模型暴露给下游消费者是一种反模式。下游消费者应该只访问使用第 3 章中所述的面向公众的数据契约格式化的数据。
Exposing the internal data model to downstream consumers is an anti-pattern. Downstream consumers should only access data formatted with public-facing data contracts as described in Chapter 3.
数据存储客户端可以在插入时对数据进行反规范化,以便发件箱反映预期的公共数据契约,尽管这确实以额外的性能和存储空间为代价。另一种选择是维护输出事件流更改的 1:1 映射,并使用专用于此任务的下游事件处理器对流进行非规范化。我将这个过程称为“事件化”,因为它将高度规范化的关系数据转换为易于使用的单个事件更新。这模仿了数据存储客户端可以执行的操作,但在数据存储外部执行此操作以减少负载。图 4-7显示了一个示例,其中根据用户、位置和雇主对用户进行非规范化。
The data store client can instead denormalize data upon insertion time such that the outbox mirrors the intended public data contract, though this does come at the expense of additional performance and storage space. Another option is to maintain the 1:1 mapping of changes to output event streams and denormalize the streams with a downstream event processor dedicated to just this task. This is a process that I call eventification, as it converts highly normalized relational data into easy-to-consume single event updates. This mimics what the data store client could do but does it externally to the data store to reduce load. An example of this is shown in Figure 4-7, where a User is denormalized based on User, Location, and Employer.
在此示例中,用户具有对其所居住的城市、州/省和国家/地区的外键引用,以及对其当前雇主的外键引用。用户事件的下游使用者可能只是希望在单个事件中获得有关每个用户的所有信息,而不是被迫将每个流具体化到状态存储中并使用关系工具对其进行非规范化,这是合理的。原始的标准化事件从发件箱发送到它们自己的事件流中,但这些流保存在组织其他部分的私有命名空间中(在“事件流元数据标记”中介绍),以保护内部数据模型。
In this example, the User has a foreign-key reference to the city, state/province, and country they live in, as well as a foreign-key reference to their current employer. It is reasonable that a downstream consumer of a User event may simply want everything about each user in a single event, instead of being forced to materialize each stream into a state store and use relational tooling to denormalize it. The raw, normalized events are sourced from the outboxes into their own event streams, but these streams are kept in a private namespace from the rest of the organization (covered in “Event Stream Metadata Tagging”) to protect the internal data model.
用户的事件化是通过对用户实体进行非规范化并摆脱任何内部数据模型结构来执行的。此过程需要维护用户、位置和雇主的物化表,以便任何更新都可以重新执行连接逻辑并为所有受影响的用户发出更新。最终事件被发送到组织的公共命名空间,供任何下游消费者使用。
Eventification of the user is performed by denormalizing the User entity and shedding any internal data model structures. This process requires maintaining materialized tables of User, Location, and Employer, such that any updates can re-exercise the join logic and emit updates for all affected Users. The final event is emitted to the public namespace of the organization for any downstream consumer to consume.
内部数据模型与外部消费者的隔离程度往往成为转向事件驱动微服务的组织的争论点。隔离内部数据模型对于确保服务的解耦和独立性至关重要,并确保系统仅因新的业务需求而需要更改,而不是上游内部数据模型更改。
The extent to which the internal data models are isolated from external consumers tends to become a point of contention in organizations moving toward event-driven microservices. Isolating the internal data model is essential for ensuring decoupling and independence of services and to ensure that systems need only change due to new business requirements, and not upstream internal data-model changes.
模式序列化(以及验证)也可以内置到捕获工作流程中。这可以在事件写入发件箱表之前或之后执行。成功意味着事件可以在工作流程中继续进行,而失败可能需要手动干预以确定根本原因并避免数据丢失。
Schema serialization (and therefore, validation) can also be built into the capture workflow. This may be performed either before or after the event is written to the outbox table. Success means the event can be proceed in the workflow, whereas a failure may require manual intervention to determine the root cause and avoid data loss.
在将事务提交到发件箱表之前进行序列化可提供最有力的数据一致性保证。序列化失败将导致事务失败并回滚对内部表所做的任何更改,以确保发件箱表和内部表保持同步。该流程如图4-8所示。成功的验证将看到事件已序列化并准备好进行事件流发布。这种方法的主要优点是显着减少了内部状态和输出事件流之间的数据不一致。事件流数据被视为一等公民,发布正确的数据被认为与维护一致的内部状态同样重要。
Serializing prior to committing the transaction to the outbox table provides the strongest guarantee of data consistency. A serialization failure will cause the transaction to fail and roll back any changes made to the internal tables, ensuring that the outbox table and internal tables stay in sync. This process is shown in Figure 4-8. A successful validation will see the event serialized and ready for event stream publishing. The main advantage of this approach is that data inconsistencies between the internal state and the output event stream are significantly reduced. The event stream data is treated as a first-class citizen, and publishing correct data is considered just as important as maintaining consistent internal state.
在写入发件箱之前进行序列化还为您提供了对所有事务使用单个发件箱的选项。格式很简单,因为内容主要是具有目标输出事件流映射的序列化数据。如图 4-9所示。
Serializing before writing to the outbox also provides you with the option of using a single outbox for all transactions. The format is simple, as the content is predominantly serialized data with the target output event stream mapping. This is shown in Figure 4-9.
在发布之前进行序列化的一个缺点是,性能可能会因序列化开销而受到影响。这对于轻负载来说可能无关紧要,但对于较重负载可能会产生更重要的影响。您需要确保满足您的性能需求。
One drawback of serializing before publishing is that performance may suffer due to the serialization overhead. This may be inconsequential for light loads but could have more significant implications for heavier loads. You will need to ensure your performance needs remain met.
或者,可以在事件写入发件箱表后执行序列化,如图4-8所示。
Alternately, serialization can be performed after the event has been written to the outbox table, as is shown in Figure 4-8.
使用此策略,您通常拥有独立的发件箱,每个域模型一个,映射到相应输出事件流的公共模式。发布者进程从发件箱读取未序列化的事件,并在将其生成到输出事件流之前尝试使用关联的模式对其进行序列化。图 4-11显示了多个发件箱的示例,一个用于用户实体,一个用于帐户实体。
With this strategy you typically have independent outboxes, one for each domain model, mapped to the public schema of the corresponding output event stream. The publisher process reads the unserialized event from the outbox and attempts to serialize it with the associated schema prior to producing it to the output event stream. Figure 4-11 shows an example of multiple outboxes, one for a User entity and one for an Account entity.
序列化失败表示事件的数据不符合其定义的架构,因此无法发布。这就是写入后序列化选项变得更难以维护的地方,因为已经完成的事务会将不兼容的数据推送到发件箱表中,并且不能保证事务可以逆转。
A failure to serialize indicates that the data of the event does not comply with its defined schema and so cannot be published. This is where the serialization-after-write option becomes more difficult to maintain, as an already completed transaction will have pushed incompatible data into the outbox table, and there is no guarantee that the transaction can be reversed.
实际上,您的发件箱中通常会出现大量不可序列化的事件。很可能需要人工干预来尝试挽救一些数据,但解决问题将非常耗时且困难,甚至可能需要停机以防止出现其他问题。由于某些事件可能确实兼容并且已经发布,导致输出流中的事件顺序可能不正确,这一事实使情况变得更加复杂。
In reality, you will typically end up with a large number of unserializable events in your outbox. Human intervention will most likely be required to try to salvage some of the data, but resolving the issue will be time-consuming and difficult and may even require downtime to prevent additional issues. This is compounded by the fact that some events may indeed be compatible and have already been published, leading to possible incorrect ordering of events in output streams.
事前序列化比事后序列化提供了针对不兼容数据的更强有力的保证,并防止违反数据契约的事件传播。代价是,如果序列化失败,此实现还将阻止业务流程完成,因为必须回滚事务。
Before-the-fact serialization provides a stronger guarantee against incompatible data than after-the-fact and prevents propagation of events that violate their data contract. The tradeoff is that this implementation will also prevent the business process from completing should serialization fail, as the transaction must be rolled back.
在写入之前进行验证和序列化可确保数据被视为一等公民,并保证输出事件流中的事件最终与源数据存储中的数据一致,同时还保持源内部的隔离数据模型。这是变更数据捕获解决方案可以提供的最有力的保证。
Validating and serializing before writing ensures that the data is being treated as a first-class citizen and offers a guarantee that events in the output event stream are eventually consistent with the data inside the source data store, while also preserving the isolation of the source’s internal data model. This is the strongest guarantee that a change-data capture solution can offer.
通过发件箱表生成事件具有许多显着的优点:
Producing events via outbox tables allow for a number of significant advantages:
任何公开事务功能的客户端或框架都支持此方法。
This approach is supported by any client or framework that exposes transactional capabilities.
在将模式插入发件箱表之前,可以通过序列化来验证模式。
Schemas can be validated by serialization before being inserted into the outbox table.
数据存储应用程序开发人员可以选择将哪些字段写入发件箱表,从而保持内部字段隔离。
Data store application developers can select which fields to write to the outbox table, keeping internal fields isolated.
在将数据写入发件箱表之前,可以根据需要对数据进行非规范化。
Data can be denormalized as needed before being written to the outbox table.
通过发件箱表生成事件也有几个缺点:
Producing events via outbox tables has several disadvantages as well:
必须更改应用程序代码才能启用此模式,这需要应用程序维护人员提供开发和测试资源。
The application code must be changed to enable this pattern, which requires development and testing resources from the application maintainers.
对业务工作流程的性能影响可能不小,特别是在通过序列化验证模式时。失败的交易还会阻止业务运营继续进行。
The performance impact to the business workflow may be nontrivial, particularly when validating schemas via serialization. Failed transactions can also prevent business operations from proceeding.
对数据存储的性能影响可能不小,尤其是在从发件箱写入、读取和删除大量记录时 。
The performance impact to the data store may be nontrivial, especially when a significant quantity of records are being written, read, and deleted from the outbox.
性能影响必须与其他成本相平衡。例如,一些组织只是通过解析变更数据捕获日志来发出事件,并让下游团队在事后清理事件。这会产生一系列费用,其形式为处理和标准化事件的计算成本,以及解决不兼容模式和关注与内部数据模型强耦合的影响形式的人力成本。生产者方面节省的成本往往与消费者方面处理这些问题所产生的费用相形见绌。
Performance impacts must be balanced against other costs. For instance, some organizations simply emit events by parsing change-data capture logs and leave it up to downstream teams to clean up the events after the fact. This incurs its own set of expenses in the form of computing costs for processing and standardizing the events, as well as human-labor costs in the form of resolving incompatible schemas and attending to the effects of strong coupling to internal data models. Costs saved at the producer side are often dwarfed by the expenses incurred at the consumer side for dealing with these issues.
触发器支持早于前面几节中讨论的许多审核、二进制日志和预写日志模式。许多旧的关系数据库使用触发器作为生成审计表的手段。顾名思义,触发器被设置为在特定条件下自动发生。如果失败,导致触发器执行的命令也会失败,从而确保更新原子性。
Trigger support predates many of the auditing, binlog, and write-ahead log patterns examined in the previous sections. Many older relational databases use triggers as a means of generating audit tables. As their name implies, triggers are set up to occur automatically on a particular condition. If it fails, the command that caused the trigger to execute also fails, ensuring update atomicity.
您可以使用AFTER触发器捕获对审计表的行级更改。例如,在任何INSERT、UPDATE、 或DELETE命令之后,触发器会将相应的行写入更改数据表。这确保了对特定表所做的更改得到相应的跟踪。
You can capture row-level changes to an audit table by using an AFTER trigger. For example, after any INSERT, UPDATE, or DELETE command, the trigger will write a corresponding row to the change-data table. This ensures that changes made to a specific table are tracked accordingly.
考虑图 4-12中所示的示例。用户数据被更新到用户表中,触发器在事件发生时捕获事件。请注意,触发器还捕获插入发生的时间以及供事件发布进程使用的自动递增序列 ID。
Consider the example shown in Figure 4-12. User data is being upserted to a user table, with a trigger capturing the events as they occur. Note that the trigger is also capturing the time at which the insertion occurred as well as an autoincrementing sequence ID for the event publisher process to use.
在触发器执行期间,您通常无法使用事件模式验证更改数据,尽管这并非不可能。一个主要问题是它可能根本不受支持,因为触发器在数据库本身内执行,并且许多触发器仅限于它们可以支持的语言形式。虽然 PostgreSQL 支持 C、Python 和 Perl,可用于编写用户定义的函数来执行模式验证,但许多其他数据库不提供多语言支持。最后,即使支持触发器,它也可能太昂贵。每个触发器独立触发,需要大量的开销来存储必要的数据、模式和验证逻辑,并且对于许多系统负载来说,成本太高。
You generally cannot validate the change-data with the event schema during the execution of a trigger, though it is not impossible. One main issue is that it may simply not be supported, as triggers execute within the database itself, and many are limited to the forms of language they can support. While PostgreSQL supports C, Python, and Perl, which may be used to write user-defined functions to perform schema validation, many other databases do not provide multilanguage support. Finally, even if a trigger is supported, it may simply be too expensive. Each trigger fires independently and requires a nontrivial amount of overhead to store the necessary data, schemas, and validation logic, and for many system loads the cost is too high.
图 4-13显示了上一个示例的延续。对变更数据执行事后验证和序列化,并将成功验证的数据生成到输出事件流。不成功的数据需要根据业务需求进行错误处理,但可能需要人工 干预。
Figure 4-13 shows a continuation of the previous example. After-the-fact validation and serialization is performed on the change-data, with successfully validated data produced to the output event stream. Unsuccessful data would need to be error-handled according to business requirements, but would likely require human intervention.
变更数据捕获表模式是内部表模式和输出事件流模式之间的桥梁。这三者之间的兼容性对于确保可以将数据生成到输出事件流至关重要。由于在触发器执行期间通常不执行输出模式验证,因此最好使更改数据表与输出事件模式的格式保持同步。
The change-data capture table schema is the bridge between the internal table schema and the output event stream schema. Compatibility among all three is essential for ensuring that data can be produced to the output event stream. Because output schema validation is typically not performed during trigger execution, it is best to keep the change-data table in sync with the format of the output event schema.
在测试期间将输出事件模式的格式与更改数据表进行比较。这可能会在生产部署之前暴露不兼容性。
Compare the format of the output event schema with the change-data table during testing. This can expose incompatibilities before production deployment.
话虽这么说,触发器在许多遗留系统中都可以很好地工作。根据定义,遗留系统倾向于使用旧技术;触发器已经存在了很长一段时间,并且很可能能够提供必要的变更数据捕获机制。访问和负载模式往往是明确定义且稳定的,因此可以准确估计添加触发的影响。最后,虽然模式验证不太可能在触发过程本身期间发生,但模式本身也同样不太可能发生改变,这仅仅是由于系统的遗留性质。仅当模式预计会频繁更改时,事后验证才成为一个问题。
That being said, triggers can work great in many legacy systems. Legacy systems tend to use, by definition, old technology; triggers have existed for a very long time and may very well be able to provide the necessary change-data capture mechanism. The access and load patterns tend to be well defined and stable, such that the impact of adding triggering can be accurately estimated. Finally, although schema validation is unlikely to occur during the triggering process itself, it may be equally unlikely that the schemas themselves are going to change, simply due to the legacy nature of the system. After-the-fact validation is only an issue if schemas are expected to change frequently.
如果您可以使用更现代的功能来生成或访问更改数据,请尽量避免使用触发器。您不应该低估基于触发器的解决方案所需的开销性能和管理,特别是当涉及数十或数百个表和数据模型时。
Try to avoid the use of triggers if you can instead use more modern functionality for generating or accessing change-data. You should not underestimate the overhead performance and management required for a trigger-based solution, particularly when many dozens or hundreds of tables and data models are involved.
使用触发器的好处包括:
Benefits of using triggers include the following:
大多数关系数据库都存在触发器。
Triggers exist for most relational databases.
对于少量数据集来说,维护和配置相当容易。
Maintenance and configuration is fairly easy for a small number of data sets.
可以自定义触发器代码以仅公开特定字段的子集。这可以对向下游消费者公开哪些数据提供一定程度的隔离。
Trigger code can be customized to expose only a subset of specific fields. This can provide some isolation into what data is exposed to downstream consumers.
使用触发器的一些缺点是:
Some cons of using triggers are:
触发器与数据库表上的操作内联执行,并且会消耗大量的处理资源。根据服务的性能要求和 SLA,此方法可能会导致不可接受的负载。
Triggers execute inline with actions on the database tables and can consume non-trivial processing resources. Depending on the performance requirements and SLAs of your services, this approach may cause an unacceptable load.
对应用程序代码和数据集定义的更改可能需要相应的触发器修改。系统维护人员可能会忽略对底层触发器的必要修改,从而导致数据解放结果与内部数据集不一致。应进行全面的测试,以确保触发器工作流程按预期运行。
Changes to application code and to data set definitions may require corresponding trigger modifications. Necessary modifications to underlying triggers may be overlooked by the system maintainers, leading to data liberation results that are inconsistent with the internal data sets. Comprehensive testing should be performed to ensure the trigger workflows operate as per expectations.
所需触发器的数量与要捕获的数据集的数量成线性比例。这排除了业务逻辑中可能已存在的任何其他触发器,例如用于强制表之间的依赖关系的触发器。
The quantity of triggers required scales linearly with the number of data sets to be captured. This excludes any additional triggers that may already exist in the business logic, such as those used for enforcing dependencies between tables.
仅当记录发布到发件箱表后,才会发生输出事件的架构强制执行。这可能会导致发件箱表中出现无法发布的事件。
Schema enforcement for the output event occurs only after the record has been published to the outbox table. This can lead to unpublishable events in the outbox table.
某些数据库允许使用可以在触发器执行期间验证与输出事件模式的兼容性的语言来执行触发器(例如,Python for PostgreSQL)。这可能会增加复杂性和费用,但会显着降低下游模式不兼容的风险。
Some databases allow for triggers to be executed with languages that can validate compatibility with output event schemas during the trigger’s execution (e.g., Python for PostgreSQL). This can increase the complexity and expense, but significantly reduces the risk of downstream schema incompatibilities.
在数据解放框架中集成数据定义更改可能很困难。数据迁移是很多关系型数据库应用的常见操作,需要捕获的支持。关系数据库的数据定义更改可以包括添加、删除和重命名列;更改列的类型;以及添加或删除默认值。虽然所有这些操作都是有效的数据集更改,但它们可能会为释放事件流的数据生成带来问题。
Integrating data definition changes can be difficult in a data liberation framework. Data migrations are a common operation for many relational database applications and need to be supported by capture. Data definition changes for a relational database can include adding, deleting, and renaming columns; changing the type of a column; and adding or removing defaults. While all of these operations are valid data set changes, they can create issues for the production of data to liberated event streams.
数据定义是数据集的形式化描述。例如,关系数据库中的表是使用数据定义语言(DDL)来定义的。生成的表、列、名称、类型和索引都是其数据定义的一部分。
Data definition is the formal description of the data set. For example, a table in a relational database is defined using a data definition language (DDL). The resultant table, columns, names, types, and indices are all part of its data definition.
例如,如果需要完全模式演化兼容性,则无法从捕获的数据集中删除没有默认值的不可为空列,因为使用先前定义的模式的使用者期望该字段的值。消费者将无法退回到任何默认状态,因为在合同定义时没有指定任何默认值,因此他们最终会处于不明确的状态。如果不兼容的更改是绝对必要的并且违反数据合同是不可避免的,那么数据的生产者和消费者必须同意新的数据合同。
For example, if full schema evolution compatibility is required, you cannot drop a non-nullable column without a default value from the data set under capture, as consumers using the previously defined schema expect a value for that field. Consumers would be unable to fall back to any default because none was specified at contract definition time, so they would end up in an ambiguous state. If an incompatible change is absolutely necessary and a breach of data contract is inevitable, then the producer and consumers of the data must agree upon a new data contract.
对捕获的数据集的有效更改可能不是对释放的事件模式的有效更改。这种不兼容性将导致破坏性架构更改,从而影响事件流的所有下游使用者。
Valid alterations to the data set under capture may not be valid changes for the liberated event schema. This incompatibility will cause breaking schema changes that will impact all downstream consumers of the event stream.
捕获 DDL 更改取决于用于捕获更改数据的集成模式。由于 DDL 更改可能会对数据的下游使用者产生重大影响,因此确定您的捕获模式是否在事前或事后检测到 DDL 的更改非常重要。例如,查询模式和 CDC 日志模式只能在事后检测 DDL 更改,即一旦它们已经应用于数据集。相反,变更数据表模式与源系统的开发周期集成,使得对数据集所做的变更需要在生产发布之前使用变更数据表进行验证。
Capturing DDL changes depends on the integration pattern used to capture change-data. As DDL changes can have a significant impact on downstream consumers of the data, it’s important to determine if your capture patterns detect changes to the DDL before or after the fact. For instance, the query pattern and CDC log pattern can detect DDL changes only after the fact—that is, once they have already been applied to the data set. Conversely, the change-data table pattern is integrated with the development cycle of the source system, such that changes made to the data set require validation with the change-data table prior to production release.
对于查询模式,可以在查询时获得模式,并且可以推断出事件模式。新的事件模式可以与输出事件流模式进行比较,模式兼容性规则用于允许或禁止发布事件数据。许多查询连接器都使用这种模式生成机制,例如Kafka Connect 框架提供的连接器。
For the query pattern, the schema can be obtained at query time, and an event schema can be inferred. The new event schema can be compared with the output event stream schema, with schema compatibility rules used to permit or prohibit publishing of the event data. This mechanism of schema generation is used by numerous query connectors, such as those provided with the Kafka Connect framework.
对于 CDC 日志模式,数据定义更新通常被捕获到 CDC 日志中自己的部分。这些更改需要从日志中提取并推断为代表数据集的模式。生成架构后,可以根据下游事件架构对其进行验证。然而,对此功能的支持是有限的。目前,Debezium 连接器仅支持 MySQL 的数据定义更改。
For the CDC log pattern, data definition updates are typically captured to their own part of the CDC log. These changes need to be extracted from the logs and inferred into a schema representative of the data set. Once the schema is generated, it can be validated against the downstream event schema. Support for this functionality, however, is limited. Currently, the Debezium connector supports only MySQL’s data definition changes.
更改数据表充当输出事件流模式和内部状态模式之间的桥梁。应用程序的验证代码或数据库的触发函数中的任何不兼容都将阻止数据写入更改数据表,并将错误发送回堆栈。根据其模式兼容性规则,对变更数据捕获表进行的更改将需要与输出事件流兼容的模式演变。这涉及一个两步过程,可显着降低意外更改进入生产的机会。
The change-data table acts as a bridge between the output event stream schema and the internal state schema. Any incompatibilities in the application’s validation code or the database’s trigger function will prevent the data from being written to the change-data table, with the error sent back up the stack. Alterations made to the change-data capture table will require a schema evolution compatible with the output event stream, according to its schema compatibility rules. This involves a two-step process, which significantly reduces the chance of unintentional changes finding their way into production.
从事件流中接收数据包括使用事件数据并将其插入数据存储中。这是通过集中式框架或独立的微服务来促进的。任何类型的事件数据,无论是实体、键控事件还是非键控事件,都可以存储到数据存储中。
Sinking data from event streams consists of consuming event data and inserting it into a data store. This is facilitated either by the centralized framework or by a standalone microservice. Any type of event data, be it entity, keyed events, or unkeyed events, can be sunk to a data store.
事件接收对于将非事件驱动的应用程序与事件流集成特别有用。接收器进程从事件代理读取事件流并将数据插入到指定的数据存储中。它跟踪自己的消耗偏移量,并在事件数据到达输入时写入事件数据,完全独立于非事件驱动的应用程序。
Event sinking is particularly useful for integrating non-event-driven applications with event streams. The sink process reads the event streams from the event broker and inserts the data into the specified data store. It keeps track of its own consumption offsets and writes event data as it arrives at the input, acting completely independently of the non-event-driven application.
事件接收的典型用途是取代遗留系统之间的直接点对点耦合。一旦源系统的数据被释放到事件流中,只需很少的其他更改就可以将其下沉到目标系统。接收器进程在外部运行,并且对于目标系统来说是不可见的。
A typical use of event sinking is replacing direct point-to-point couplings between legacy systems. Once the data of the source system is liberated into event streams, it can be sunk to the destination system with few other changes. The sink process operates both externally and invisibly to the destination system.
需要执行基于批量的大数据分析的团队也经常使用数据下沉。他们通常通过将数据下沉到 Hadoop 分布式文件系统来实现这一点,该系统提供大数据分析工具。
Data sinking is also employed frequently by teams that need to perform batch-based big-data analysis. They usually do this by sinking data to a Hadoop Distributed File System, which provides big-data analysis tools.
使用像 Kafka Connect 这样的通用平台,您可以通过简单的配置来指定接收器,并在共享基础设施上运行它们。独立的微服务接收器提供了另一种解决方案。开发人员可以在微服务平台上创建和运行它们,并独立管理它们。
Using a common platform like Kafka Connect allows you to specify sinks with simple configurations and run them on the shared infrastructure. Standalone microservice sinks provide an alternative solution. Developers can create and run them on the microservice platform and manage them independently.
集中式框架允许以较低的开销流程来释放数据。该框架可以由单个团队大规模运营,从而支持整个组织中其他团队的数据解放需求。寻求集成的团队只需关注连接器配置和设计,而不需要任何操作职责。这种方法最适合大型组织,其中数据存储在多个团队的多个数据存储中,因为它允许快速开始数据解放,而不需要每个团队构建自己的解决方案。
A centralized framework allows for lower-overhead processes for liberating data. This framework may be operated at scale by a single team, which in turn supports the data liberation needs of other teams across the organization. Teams looking to integrate then need only concern themselves with the connector configuration and design, not with any operational duties. This approach works best in larger organizations where data is stored in multiple data stores across multiple teams, as it allows for a quick start to data liberation without each team needing to construct its own solution.
使用集中式框架时,您可能会陷入两个主要陷阱。首先,数据源/汇的职责现在在团队之间共享。操作集中式框架的团队负责框架和每个连接器实例的稳定性、扩展性和运行状况。同时,操作系统的捕获系统团队是独立的,并且可能会做出改变连接器性能和稳定性的决策,例如添加和删除字段,或更改影响通过连接器传输的数据量的逻辑。这引入了这两个团队之间的直接依赖关系。这些更改可能会破坏连接器,但可能只能由连接器管理团队检测到,从而导致线性扩展、跨团队依赖性。
There are two main traps that you can fall into when using a centralized framework. First, the data sourcing/sinking responsibilities are now shared between teams. The team operating the centralized framework is responsible for the stability, scaling, and health of both the framework and each connector instance. Meanwhile, the team operating the system under capture is independent and may make decisions that alter the performance and stability of the connector, such as adding and removing fields, or changing logic that affects the volume of data being transmitted through the connector. This introduces a direct dependency between these two teams. These changes can break the connectors, but may be detected only by the connector management team, leading to linearly scaling, cross-team dependencies. This can become a difficult-to-manage burden as the number of changes grows.
第二个问题更为普遍,尤其是在仅部分采用事件驱动原则的组织中。系统可能会过于依赖框架和连接器来完成事件驱动的工作。一旦数据从内部状态存储中解放出来并发布到事件流,组织可能会因为转向微服务而沾沾自喜。团队可能会过度依赖连接器框架来获取和接收数据,并且选择不将其应用程序重构为本机事件驱动应用程序。在这种情况下,他们更愿意根据需要仅申请新的源和接收器,从而使整个底层应用程序完全不了解事件。
The second issue is a bit more pervasive, especially in an organization where event-driven principles are only partially adopted. Systems can become too reliant upon frameworks and connectors to do their event-driven work for them. Once data has been liberated from the internal state stores and published to event streams, the organization may become complacent about moving onward into microservices. Teams can become overly reliant upon the connector framework for sourcing and sinking data, and choose not to refactor their applications into native event-driven applications. In this scenario they instead prefer to just requisition new sources and sinks as necessary, leaving their entire underlying application completely ignorant to events.
CDC 工具并不是迁移到事件驱动架构的最终目的地,而是主要用于帮助引导该流程。事件代理作为数据通信层的真正价值在于提供与实现层解耦的稳健、可靠和真实的事件数据源,并且代理的好坏取决于其数据的质量和可靠性。
CDC tools are not the final destination in moving to an event-driven architecture, but instead are primarily meant to help bootstrap the process. The real value of the event broker as the data communication layer is in providing a robust, reliable, and truthful source of event data decoupled from the implementation layers, and the broker is only as good as the quality and reliability of its data.
通过正确理解变更数据捕获框架的作用,这两个问题都可以得到缓解。也许与直觉相反,重要的是尽量减少 CDC 框架的使用,并让团队实现自己的变更数据捕获(例如发件箱模式),尽管这可能需要额外的前期工作。团队完全负责发布及其系统事件,消除了跨团队依赖性和基于连接器的脆弱的 CDC。这最大限度地减少了 CDC 框架团队需要完成的工作,并使他们能够专注于支持真正需要它的产品。
Both of these issues can be mitigated through a proper understanding of the role of the change-data capture framework. Perhaps counterintuitively, it’s important to minimize the usage of the CDC framework and have teams implement their own change-data capture (such as the outbox pattern) despite the additional up-front work this may require. Teams become solely responsible for publishing and their system’s events, eliminating cross-team dependencies and brittle connector-based CDC. This minimizes the work that the CDC framework team needs to do and allows them to focus on supporting products that truly need it.
减少对 CDC 框架的依赖也传播了“事件优先”的思维模式。您不要将事件流视为在单体之间重新整理数据的一种方式,而是将每个系统视为事件的直接发布者和消费者,加入事件驱动的生态系统。通过成为 EDM 生态系统的积极参与者,您开始考虑系统需要何时以及如何生成事件,考虑外面的数据而不仅仅是这里的数据。这是成功实施 EDM 文化转变的重要组成部分。
Reducing the reliance on the CDC framework also propagates an “event-first” mind-set. Instead of thinking of event streams as a way to shuffle data between monoliths, you view each system as a direct publisher and consumer of events, joining in on the event-driven ecosystem. By becoming an active participant in the EDM ecosystem, you begin to think about when and how the system needs to produce events, about the data out there instead of just the data in here. This is an important part of the cultural shift toward successful implementation of EDM.
对于资源有限的产品和仅进行维护操作的产品,集中式源和接收器连接器系统可能会带来重大好处。对于其他产品,尤其是那些更复杂、具有重要事件流要求并且正在积极开发的产品,对连接器的持续维护和支持是不可持续的。在这些情况下,最好根据需要安排时间重构代码库,以使应用程序成为真正的本机事件驱动应用程序。
For products with limited resources and those under maintenance-only operation, a centralized source and sink connector system can be a significant boon. For other products, especially those that are more complex, have significant event stream requirements, and are under active development, ongoing maintenance and support of connectors is unsustainable. In these circumstances it is best to schedule time to refactor the codebase as necessary to allow the application to become a truly native event-driven application.
最后,仔细考虑每种 CDC 策略的权衡。这通常成为组织内讨论和争论的一个领域,因为团队试图找出他们在将事件作为唯一事实来源制作事件方面的新责任和界限。转向事件驱动的架构需要对数据通信层进行投资,并且该层的有用性取决于其内部数据的质量。组织内的每个人都必须转变思维,考虑其释放的数据对组织其他部分的影响,并就其所处理事件的模式、数据模型、排序、延迟和正确性提出明确的服务级别协议生产。
Finally, carefully consider the tradeoffs of each of the CDC strategies. This often becomes an area of discussion and contention within an organization, as teams try to figure out their new responsibilities and boundaries in regard to producing their events as the single source of truth. Moving to an event-driven architecture requires investment into the data communication layer, and the usefulness of this layer can only ever be as good as the quality of data within it. Everyone within the organization must shift their thinking to consider the impacts of their liberated data on the rest of the organization and come up with clear service-level agreements as to the schemas, data models, ordering, latency, and correctness for the events they are producing.
数据解放是提供成熟且可访问的数据通信层的重要一步。遗留系统通常包含大量核心业务领域模型,存储在某种形式的集中式实施通信结构中。这些数据需要从这些遗留系统中解放出来,以使组织的其他领域能够组成新的、解耦的产品和服务。
Data liberation is an important step toward providing a mature and accessible data communication layer. Legacy systems frequently contain the bulk of the core business domain models, stored within some form of centralized implementation communication structure. This data needs to be liberated from these legacy systems to enable other areas of the organization to compose new, decoupled products and services.
有许多框架、工具和策略可用于从其实施数据存储中提取和转换数据。每种方法都有自己的优点、缺点和权衡。您的用例将影响您选择的选项,或者您可能会发现必须创建自己的机制和流程。
There are a number of frameworks, tools, and strategies available to extract and transform data from their implementation data stores. Each has its own benefits, drawbacks, and tradeoffs. Your use cases will influence which options you select, or you may find that you must create your own mechanisms and processes.
数据解放的目标是为对组织重要的数据提供干净且一致的单一事实来源。数据访问与数据的生成和存储分离,从而无需实现通信结构来承担双重任务。这个简单的行为减少了从遗留系统的众多实现中访问重要领域数据的界限,并直接促进新产品和服务的开发。
The goal of data liberation is to provide a clean and consistent single source of truth for data important to the organization. Access to data is decoupled from the production and storage of it, eliminating the need for implementation communication structures to serve double duty. This simple act reduces the boundaries for accessing important domain data from the numerous implementations of legacy systems and directly promotes the development of new products and services.
有各种各样的数据解放策略。一方面,您会发现与源系统的仔细集成,其中事件在写入实现数据存储时被发送到事件代理。有些系统甚至可以先生成事件流,然后再将其消费回以满足自己的需要,从而进一步强化事件流作为单一事实来源的地位。生产者认识到自己作为良好的数据生产公民的角色,并采取保护措施以防止意外的破坏性更改。生产者寻求与消费者合作,以确保高质量、定义明确的数据流,最大限度地减少破坏性更改,并确保对系统的更改与他们正在生成的事件的模式兼容。
There is a full spectrum of data liberation strategies. On one end you will find careful integration with the source system, where events are emitted to the event broker as they are written to the implementation data store. Some systems may even be able to produce to the event stream first before consuming it back for their own needs, further reinforcing the event stream as the single source of truth. The producer is cognizant of its role as a good data-producing citizen and puts protections in place to prevent unintentional breaking changes. Producers seek to work with the consumers to ensure a high-quality, well-defined data stream, minimize disruptive changes, and ensure changes to the system are compatible with the schemas of the events they are producing.
另一方面,您会发现高度反应性的策略。实现中源数据的所有者对事件代理中数据的生成几乎没有可见性。他们完全依赖框架直接从内部数据集中提取数据或解析变更数据捕获日志。破坏下游消费者的损坏模式很常见,源实现中内部数据模型的暴露也是如此。从长远来看,这种模型是不可持续的,因为它忽略了数据所有者确保领域事件的干净、一致的生成的责任。
On the other end of the spectrum, you’ll find the highly reactive strategies. The owners of the source data in the implementation have little to no visibility into the production of data into the event broker. They rely completely on frameworks to either pull the data directly from their internal data sets or parse the change-data capture logs. Broken schemas that disrupt downstream consumers are common, as is exposure of internal data models from the source implementation. This model is unsustainable in the long run, as it neglects the responsibility of the owner of the data to ensure clean, consistent production of domain events.
组织的文化决定了数据解放计划在转向事件驱动架构方面的成功程度。数据所有者必须认真对待生成干净可靠的事件流的需求,并了解数据捕获机制不足以作为释放事件数据的最终目的地。
The culture of the organization dictates how successful data liberation initiatives will be in moving toward an event-driven architecture. Data owners must take seriously the need to produce clean and reliable event streams, and understand that data capture mechanisms are insufficient as a final destination for liberating event data.
Most event-driven microservices follow, at a minimum, the same three steps:
使用输入事件流中的事件。
Consume an event from an input event stream.
处理该事件。
Process that event.
产生任何必要的输出事件。
Produce any necessary output events.
还有事件驱动的微服务,它们从同步请求-响应交互中派生输入事件,第 13 章对此进行了详细介绍。本章仅涵盖从事件流获取事件的微服务。
There are also event-driven microservices that derive their input event from a synchronous request-response interaction, which is covered more in Chapter 13. This chapter covers only microservices that source their events from event streams.
在流源事件驱动的微服务中,微服务实例将创建一个生产者客户端和一个消费者客户端,并向任何必要的消费者组注册自身(如果适用)。微服务启动一个循环来轮询消费者客户端是否有新事件,在它们进入时对其进行处理并发出任何所需的输出事件。该工作流程如以下伪代码所示。(您的实现当然会根据您的语言、流处理框架、事件代理选择和其他技术因素而有所不同。)
In stream-sourced event-driven microservices, the microservice instance will create a producer client and a consumer client and register itself with any necessary consumer groups, if applicable. The microservice starts a loop to poll the consumer client for new events, processing them as they come in and emitting any required output events. This workflow is shown in the following pseudocode. (Your implementation will of course vary according to your language, stream-processing framework, event-broker selection, and other technical factors.)
ConsumerconsumerClient=newconsumerClient(consumerGroupName,...);ProducerproducerClient=newproducerClient(...);while(true){InputEventevent=consumerClient.pollOneEvent(inputEventStream);OutputEventoutput=processEvent(event);producerClient.produceEventToStream(outputEventStream,output);//At-least-once processing.consumerClient.commitOffsets();}
ConsumerconsumerClient=newconsumerClient(consumerGroupName,...);ProducerproducerClient=newproducerClient(...);while(true){InputEventevent=consumerClient.pollOneEvent(inputEventStream);OutputEventoutput=processEvent(event);producerClient.produceEventToStream(outputEventStream,output);//At-least-once processing.consumerClient.commitOffsets();}
该processEvent函数特别令人感兴趣。这是真正的事件处理工作完成的地方,主要是业务逻辑的应用以及要发出的事件(如果有)。最好将此处理函数视为微服务处理拓扑的入口点。从这里开始,数据驱动模式将转换和处理数据以满足有界上下文的业务需求。
The processEvent function is of particular interest. This is where the real event-processing work gets done, primarily the application of business logic and which events, if any, to emit. This processing function is best thought of as the entry point to the processing topology of the microservice. From here, data-driven patterns transform and process the data for your bounded context’s business needs.
构建微服务拓扑需要以事件驱动的方式进行思考,因为代码会响应到达消费者输入的事件而执行。微服务的拓扑本质上是对事件执行的一系列操作。它需要选择必要的过滤器、路由器、转换、物化、聚合以及执行微服务必要的业务逻辑所需的其他功能。熟悉函数式编程和大数据 Map-Reduce 风格框架的人可能会在这里感到很自在。对于其他人来说,这可能是一个新概念。
Building a microservice topology requires thinking in an event-driven way, as the code executes in response to an event arriving at the consumer input. The topology of the microservice is essentially a sequence of operations to perform on the event. It requires choosing the necessary filters, routers, transformations, materializations, aggregations, and other functions required to perform the necessary business logic of the microservice. Those familiar with functional programming and big data map-reduce-style frameworks may feel quite at home here. For others, this may be a bit of a new concept.
考虑图 5-1中的拓扑。事件一次消耗一个,并根据阶段 1 和阶段 2 中的转换进行处理。
Consider the topology in Figure 5-1. Events are consumed one at a time and are processed according to the transformations in stages 1 and 2.
A带有 key和 的事件C都会遍历整个拓扑。它们都大于10.0,这使它们能够通过第 1 阶段,而第 2 阶段只是删除事件值的小数点。B然而,键入 的事件会被过滤掉并丢弃,因为它不符合第 1 阶段的标准。
The events with key A and C both traverse the entire topology. They’re both larger than 10.0, which gets them through stage 1, and stage 2 simply drops the decimal point from the value of the event. The event keyed on B, however, is filtered out and dropped because it does not meet the stage 1 criteria.
转换处理单个事件并发出零个或多个输出事件。正如您可能猜到的那样,转换提供了需要转换的大量业务逻辑操作。根据操作,事件可能需要重新分区(稍后将详细介绍)。常见的转换包括但不限于以下内容:
A transform processes a single event and emits zero or more output events. Transforms, as you might guess, provide the bulk of the business logic operations requiring transformations. Events may need to be repartitioned depending on the operations (more on this shortly). Common transformations include, but are not limited to, the following:
如果满足必要的标准,则传播该事件。发出零个或一个事件。
Propagate the event if it meets the necessary criteria. Emits zero or one events.
更改事件的键和/或值,仅发出一个事件。请注意,如果更改密钥,可能需要重新分区以确保数据局部性。
Changes the key and/or value of the event, emitting exactly one event. Note that if you change the key, you may need to repartition to ensure data locality.
仅更改事件的值,而不更改键。只发出一个事件。不需要重新分区。
Change only the value of the event, not the key. Emits exactly one event. Repartitioning will not be required.
应用自定义逻辑、查找状态,甚至与其他系统同步通信。
Apply custom logic, look up state, and even communicate with other systems synchronously.
消费者应用程序可能需要对事件流进行分支,即将逻辑运算符应用于事件,然后根据结果将其输出到新的流。一种相对常见的场景是使用事件的“消防水带”,并根据特定属性(例如,国家、时区、来源、产品或任意数量的特征)决定将它们路由到何处。第二种常见场景是将结果发送到不同的输出事件流,例如,在发生处理错误时将事件输出到死信流,而不是完全丢弃它们。
A consumer application may need to branch event streams—that is, apply a logical operator to an event and then output it to a new stream based on the result. One relatively common scenario is consuming a “firehose” of events and deciding where to route them based on particular properties (e.g., country, time zone, origin, product, or any number of features). A second common scenario is emitting results to different output event streams—for example, outputting events to a dead-letter stream in case of a processing error, instead of dropping them completely.
应用程序可能还需要合并流,其中使用来自多个输入流的事件,可能以某种有意义的方式进行处理,然后输出到单个输出流。在很多情况下,将多个流合并为一个流很重要,因为微服务通常会根据需要消耗尽可能多的输入流来实现其业务逻辑。第 6 章讨论如何以一致且可重现的顺序处理来自多个输入流的消费和处理事件。
Applications may also need to merge streams, where events from multiple input streams are consumed, possibly processed in some meaningful way, and then output to a single output stream. There aren’t too many scenarios where it’s important to merge multiple streams into just one, since it is common for microservices to consume from as many input streams as necessary to fulfill their business logic. Chapter 6 discusses how to handle consuming and processing events from multiple input streams in a consistent and reproducible order.
如果您最终合并了事件流,请定义一个新的统一模式来代表合并的事件流域。如果此域没有意义,那么最好不要合并流并重新考虑您的系统设计。
If you do end up merging event streams, define a new unified schema representative of the merged event steam domain. If this domain doesn’t make sense, then it may be best to leave the streams unmerged and reconsider your system design.
事件流根据事件键和事件分区器逻辑进行分区。对于每个事件,应用事件分区器,并为要写入的事件选择一个分区。重新分区是生成具有以下一个或多个属性的新事件流的行为:
Event streams are partitioned according to the event key and the event partitioner logic. For each event, the event partitioner is applied, and a partition is selected for the event to be written to. Repartitioning is the act of producing a new event stream with one or more of the following properties:
Increase an event stream’s partition count to increase downstream parallelism or to match the number of partitions of another stream for copartitioning (covered later in this chapter).
更改事件键以确保具有相同键的事件路由到同一分区。
Change the event key to ensure that events with the same key are routed to the same partition.
更改用于选择将事件写入哪个分区的逻辑。
Change the logic used to select which partition an event will be written to.
纯粹的无状态处理器很少需要重新分区事件流,除非增加分区计数以增加下游并行性。话虽这么说,无状态微服务可用于重新分区下游有状态处理器消耗的事件,这是下一个示例的主题 。
It’s rare that a purely stateless processor will need to repartition an event stream, barring the case of increasing the partition count for increased downstream parallelism. That being said, a stateless microservice may be used to repartition events that are consumed by a downstream stateful processor, which is the subject of the next example.
分区器算法通常使用哈希函数确定性地将事件的键映射到特定分区。这可确保具有相同键的所有事件最终位于同一分区中。
The partitioner algorithm deterministically maps an event’s key to a specific partition, typically by using a hash function. This ensures that all events with the same key end up in the same partition.
假设有来自面向 Web 的端点的用户数据流。用户操作被转换为事件,事件的有效负载包含用户 ID 和其他任意事件数据,标记为x。
Suppose there is a stream of user data coming in from a web-facing endpoint. The user actions are converted into events, with the payload of the events containing both a user ID and other arbitrary event data, labeled x.
此状态的使用者有兴趣确保属于特定用户的所有数据都包含在同一分区中,无论源事件流如何分区。可以对该流重新分区以确保这种情况,如图5-2所示。
Consumers of this state are interested in ensuring that all of the data belonging to a particular user is contained within the same partition, regardless of how the source event stream is partitioned. This stream can be repartitioned to ensure this is the case, as shown in Figure 5-2.
将给定键的所有事件生成到单个分区中为数据局部性提供了基础。消费者只需要使用来自单个分区的事件来构建与该键相关的事件的完整图片。这使得消费者微服务能够扩展到多个实例,每个实例都从单个分区进行消费,同时维护与该密钥相关的所有事件的完整状态帐户。重新分区和数据局部性是大规模执行有状态处理的重要组成部分。
Producing all events for a given key into a single partition provides the basis for data locality. A consumer need only consume events from a single partition to build a complete picture of events pertaining to that key. This enables consumer microservices to scale up to many instances, each consuming from a single partition, while maintaining a complete stateful account of all events pertaining to that key. Repartitioning and data locality are essential parts of performing stateful processing at scale.
共同分区是将一个事件流重新分区为一个新事件流,该新事件流具有与另一个流相同的分区计数和分区分配器逻辑。当一个事件流中的键控事件需要与另一流的事件共置(为了数据局部性)时,这是必需的。这是有状态流处理的一个重要概念,因为许多有状态操作(例如流连接)要求给定键的所有事件,无论它们来自哪个流,都通过同一节点进行处理。第 7 章对此进行了更详细的介绍。
Copartitioning is the repartition of an event stream into a new one with the same partition count and partition assignor logic as another stream. This is required when keyed events from one event stream need to be colocated (for data locality) with the events of another stream. This is an important concept for stateful stream processing, as numerous stateful operations (such as streaming joins) require that all events for a given key, regardless of which stream they’re from, be processed through the same node. This is covered in more detail in Chapter 7.
再次考虑图 5-2的重新分区示例。假设您现在需要将重新分区的用户事件流与用户实体流连接起来,并以相同的 ID 为关键。这些流的连接如图 5-3所示。
Consider again the repartition example of Figure 5-2. Say that you now need to join the repartitioned user event stream with a user entity stream, keyed on that same ID. These joining of these streams is shown in Figure 5-3.
两个流具有相同的分区计数,并且都使用相同的分区器算法进行分区。请注意,每个分区的键分布与其他流的分布相匹配,并且每个连接都由其自己的使用者实例执行。下一节将介绍如何将分区分配给微服务实例以利用共同分区的流,如本联接示例中所做的那样。
Both streams have the same partition count, and both have been partitioned using the same partitioner algorithm. Note that the key distribution of each partition matches the distribution of the other stream and that each join is performed by its own consumer instance. The next section covers how partitions are assigned to a microservice instance to leverage copartitioned streams, as was done in this join example.
每个微服务都维护自己独特的消费者组,代表其输入事件流的集体偏移量。第一个上线的消费者实例将使用其消费者组名称向事件代理注册。注册后,需要为消费者实例分配分区。
Each microservice maintains its own unique consumer group representing the collective offsets of its input event streams. The first consumer instance that comes online will register with the event broker using its consumer group name. Once registered, the consumer instance will then need to be assigned partitions.
一些事件代理(例如 Apache Kafka)将分区分配委托给每个消费者组的第一个在线客户端。作为消费者组领导者,该实例负责执行分区分配者职责,确保每当新实例加入该消费者组时,正确分配输入事件流分区。
Some event brokers, such as Apache Kafka, delegate partition assignment to the first online client for each consumer group. As consumer group leader, this instance is responsible for performing the partition assignor duties, ensuring that input event-stream partitions are correctly assigned whenever new instances join that consumer group.
其他事件代理(例如 Apache Pulsar)在代理内维护分区分配的集中所有权。在这种情况下,分区分配和重新平衡由代理完成,但通过消费者组进行识别的机制保持不变。分区已分配,工作可以从消耗事件的最后一个已知偏移量开始。
Other event brokers, such as Apache Pulsar, maintain a centralized ownership of partition assignment within the broker. In this case, the partition assignment and rebalancing are done by the broker, but the mechanism of identification via consumer group remains the same. Partitions are assigned, and work can begin from the last known offsets of consumed events.
在重新分配分区以避免分配竞争情况时,工作通常会暂时挂起。这可确保任何已撤销的分区在分配给新实例之前不再由另一个实例处理,从而消除任何潜在的重复输出。
Work is usually momentarily suspended while partitions are reassigned to avoid assignment race conditions. This ensures that any revoked partitions are no longer being processed by another instance before assignment to the new instance, eliminating any potential duplicate output.
处理大量数据通常需要消费者微服务的多个实例,无论是专用的流处理框架还是基本的生产者/消费者实现。分区分配器确保分区以平衡且公平的方式分配给处理实例。
Multiple instances of a consumer microservice are typically required for processing large volumes of data, whether it’s a dedicated stream-processing framework or a basic producer/consumer implementation. A partition assignor ensures that partitions are distributed to the processing instances in a balanced and equitable manner.
每当在消费者组中添加或删除新的消费者实例时,该分区分配器还负责重新分配分区。根据您的事件代理选择,该组件可能内置于消费者客户端中或在事件代理中维护。
This partition assignor is also responsible for reassigning partitions whenever new consumer instances are added or removed from the consumer group. Depending on your event broker selection, this component may be built into the consumer client or maintained within the event broker.
分区分配者还负责确保满足任何共同分区要求。所有标记为共同分区的分区必须分配给同一个消费者实例。这可确保为给定的微服务实例分配正确的事件数据子集来执行其业务逻辑。最好让分区分配器实现检查事件流是否具有相等的分区计数,并在不相等时抛出异常。
The partition assignor is also responsible for ensuring that any copartitioning requirements are met. All partitions marked as copartitioned must be assigned to the same single consumer instance. This ensures that a given microservice instance will be assigned the correct subset of event data to perform its business logic. It is good practice to have the partition assignor implementation check to see that the event streams have an equal partition count and throw an exception on inequality.
分区分配算法的目标是确保分区均匀分布在消费者实例上,假设消费者实例的处理能力相同。分区分配算法还可能有次要目标,例如减少重新平衡期间重新分配的分区数量。当您处理跨多个数据存储实例分片的物化状态时,这一点尤其重要,因为分区的重新分配可能会导致将来的更新进入错误的分片。第 7 章进一步探讨了有关内部状态存储的概念。
The goal of a partition assignment algorithm is to ensure that partitions are evenly distributed across the consumer instances, assuming that the consumer instances are equal in processing capabilities. A partition assignment algorithm may also have secondary goals, such as reducing the number of partitions reassigned during a rebalance. This is particularly important when you are dealing with materialized state sharded across multiple data store instances, as the reassignment of a partition can cause future updates to go to the wrong shard. Chapter 7 explores this concept further with regard to internal state stores.
有许多常见的分配分区策略。默认策略可能会根据您的框架或实现的不同而有所不同,但以下三种策略往往是最常用的。
There are a number of common strategies for assigning partitions. The default strategy may vary depending on your framework or implementation, but the following three tend to be the most commonly used.
所有分区都记录在一个列表中,并以循环方式分配给每个消费者实例。为共同分区的流保留一个单独的列表,以确保正确的共同分区分配。
All partitions are tallied into a list and assigned in a round-robin manner to each consumer instance. A separate list is kept for copartitioned streams to ensure proper copartitioned assignment.
图 5-4显示了两个使用者实例,每个实例都有自己的一组分配的分区。与 C1 的一组相比,C0 有两组共分区分区,因为分配都是在 C0 上开始和结束的。
Figure 5-4 shows two consumer instances, each with its own set of assigned partitions. C0 has two sets of copartitioned partitions compared to one for C1, since assignment both began and ended on C0.
当给定消费者组的消费者实例数量增加时,应重新平衡分区分配,以在新添加的资源之间分散负载。图 5-5显示了添加两个以上消费者实例的效果 。
When the number of consumer instances for the given consumer group increases, partition assignments should be rebalanced to spread the load among the newly added resources. Figure 5-5 shows the effects of adding two more consumer instances.
现在,C2 被分配了共同分区的 P2,以及流 A 的 P2。另一方面,C3 仅具有流 A 中的分区 P3,因为没有其他分区可供分配。添加任何其他实例将不会导致任何额外的 并行化。
C2 is now assigned the copartitioned P2s, as well as stream A’s P2. C3, on the other hand, only has partition P3 from stream A because there are no additional partitions to assign. Adding any further instances will not result in any additional parallelization.
当必须将特定分区分配给特定消费者时,可以使用静态分配协议。当在任何给定实例上具体化大量有状态数据(通常用于内部状态存储)时,此选项最有用。当消费者实例离开消费者组时,静态分配器不会重新分配分区,而是会等待,直到丢失的消费者实例重新上线。根据实现的不同,如果原始消费者未能在指定的时间内重新加入消费者组,则无论如何都可以动态地重新分配分区。
Static assignment protocols can be used when specific partitions must be assigned to specific consumers. This option is most useful when large volumes of stateful data are materialized on any given instance, usually for internal state stores. When a consumer instance leaves the consumer group, a static assignor will not reassign the partitions, but will instead wait until the missing consumer instance comes back online. Depending on the implementation, partitions may be dynamically reassigned anyway, should the original consumer fail to rejoin the consumer group within a designated period of time.
通过利用外部信号和工具,可以根据客户的需求定制自定义分配。例如,分配可以基于输入事件流中的当前滞后,确保在所有使用者实例之间均匀分配工作。
By leveraging external signals and tooling, custom assignments can be tailored to the needs of the client. For example, assignment could be based on the current lag in the input event streams, ensuring an equal distribution of work across all of your consumer instances.
从无状态故障中恢复实际上与简单地将新实例添加到消费者组相同。无状态处理器不需要任何状态恢复,这意味着一旦分配了分区并建立了流时间,它们就可以立即返回处理事件。
Recovering from stateless failures is effectively the same as simply adding a new instance to a consumer group. Stateless processors do not require any state restoration, which means they can immediately go back to processing events as soon as they’re assigned partitions and establish their stream time.
基本的无状态事件驱动微服务使用事件、处理事件并发出任何新的后续事件。每个事件的处理都是独立于其他事件的。基本转换允许您将事件更改为更有用的格式,然后您可以使用新的分区计数将其重新分区到新键入的事件流中。具有相同键、相同分区器算法和相同分区计数的事件流被称为共同分区,这保证了给定使用者实例的数据局部性。分区分配器用于确保消费者实例之间的分区均匀分布,并且共同分区的事件流正确地共同分配。
The basic stateless event-driven microservice consumes events, processes them, and emits any new subsequent events. Each event is processed independently of the others. Basic transformations allow you to change events into more useful formats, which you can then repartition into a newly keyed event stream with a new partition count. Event streams with the same key, the same partitioner algorithm, and the same partition count are said to be copartitioned, which guarantees data locality for a given consumer instance. The partition assignor is used to ensure that partitions between consumer instances are evenly distributed and that copartitioned event streams are correctly coassigned.
共同分区和分区分配是理解有状态处理的重要概念,第 7 章将对此进行介绍。但首先,您必须考虑如何处理来自多个事件流的多个分区。无序事件、延迟事件以及选择处理事件的顺序都会对服务的设计产生重大影响。这将是下一章的主题。
Copartitioning and partition assignment are important concepts for understanding stateful processing, which is covered in Chapter 7. First, though, you must consider how to handle processing multiple partitions from multiple event streams. Out-of-order events, late events, and the order in which events are selected for processing all have a significant impact on the design of your services. This will be the topic of the next chapter.
事件驱动的微服务通常具有比上一章中介绍的拓扑更复杂的拓扑。事件是从多个事件流中消费和处理的,而需要状态处理(在下一章中介绍)来解决许多业务问题。微服务也会遇到与非微服务系统相同的故障和崩溃。近乎实时地混合使用微服务处理事件的情况并不罕见,而其他新启动的微服务则通过处理历史数据来迎头赶上。
Event-driven microservices usually have topologies that are more complex than those introduced in the previous chapter. Events are consumed and processed from multiple event streams, while stateful processing (covered in the next chapter) is required to solve many business problems. Microservices are also subject to the same faults and crashes as nonmicroservice systems. It is not uncommon to have a mixture of microservices processing events in near–real time while other, newly started microservices are catching up by processing historical data.
以下是本章讨论的三个主要问题:
Here are the three main questions addressed in this chapter:
从多个分区消费时,微服务如何选择要处理的事件顺序?
How does a microservice choose the order of events to process when consuming from multiple partitions?
微服务如何处理无序和迟到事件?
How does a microservice handle out-of-order and late-arriving events?
我们如何确保我们的微服务在近实时处理流时与从流开头处理时产生确定性结果?
How do we ensure that our microservices produce deterministic results when processing streams in near–real time versus when processing from the beginning of the streams?
我们可以通过检查时间戳、事件调度、水印和流时间以及它们如何促进确定性处理来回答这些问题。业务逻辑中的错误、错误和变化也将需要重新处理,这使得确定性结果变得非常重要。本章还探讨了无序和迟到事件如何发生、处理它们的策略以及减轻它们对我们工作流程的影响。
We can answer these questions by examining timestamps, event scheduling, watermarks, and stream times, and how they contribute to deterministic processing. Bugs, errors, and changes in business logic will also necessitate reprocessing, making deterministic results important. This chapter also explores how out-of-order and late-arriving events can occur, strategies for handling them, and mitigating their impact on our workflows.
尽管我尽了最大努力寻找一种简单明了的方法来解释关键概念,但本章的信息相当密集。在许多部分中,我将向您推荐更多资源供您自行探索,因为详细信息通常超出了本书的范围。
This chapter is fairly information-dense despite my best efforts to find a simple and concise way to explain the key concepts. There are a number of sections where I will refer you to further resources to explore on your own, as the details often go beyond the scope of this book.
事件驱动的微服务有两种主要的处理状态。它可能会近乎实时地处理事件,这是长时间运行的微服务的典型特征。或者,它可能正在处理过去的事件,以努力赶上当前的时间,这对于规模较小的服务和新服务来说很常见。
An event-driven microservice has two main processing states. It may be processing events at near–real time, which is typical of long-running microservices. Alternately, it may be processing events from the past in an effort to catch up to the present time, which is common for underscaled and new services.
如果您要将输入事件流的使用者组偏移量倒回到时间开始处并再次启动微服务运行,它会生成与第一次运行时相同的输出吗?确定性处理的首要目标是微服务应该产生相同的输出,无论是实时处理还是赶上当前时间。
If you were to rewind the consumer group offsets of the input event streams to the beginning of time and start the microservice run again, would it generate the same output as the first time it was run? The overarching goal of deterministic processing is that a microservice should produce the same output whether it is processing in real time or catching up to the present time.
请注意,有些工作流程是明确不确定的,例如基于当前挂钟时间的工作流程和查询外部服务的工作流程。外部服务可能会根据查询的时间提供不同的结果,特别是如果其内部状态的更新独立于发出查询的服务的内部状态。在这些情况下,无法保证确定性,因此请务必注意工作流程中的任何非确定性操作。
Note that there are workflows that are explicitly nondeterministic, such as those based on the current wall-clock time and those that query external services. External services may provide different results depending on when they are queried, especially if their internal state is updated independently of that from the services issuing the query. In these cases there is no promise of determinism, so be sure to pay attention to any nondeterministic operations in your workflow.
完全确定性处理是理想的情况,其中每个事件都按时到达,并且没有延迟,没有生产者或消费者故障,也没有间歇性网络问题。由于我们别无选择,只能处理这些场景,所以现实情况是我们的服务只能在确定性方面做出最大努力。有许多组件和流程协同工作来促进此尝试,并且在大多数情况下,尽力而为的确定论足以满足您的要求。要实现此目标,您需要做一些事情:一致的时间戳、精心选择的事件键、分区分配、事件调度以及处理迟到事件的策略。
Fully deterministic processing is the ideal case, where every event arrives on time and there is no latency, no producer or consumer failures, and no intermittent network issues. Since we have no choice but to deal with these scenarios, the reality is that our services can only achieve a best effort at determinism. There are a number of components and processes that work together to facilitate this attempt, and in most cases best-effort determinism will be sufficient for your requirements. There are a few things you need to achieve this: consistent timestamps, well-selected event keys, partition assignment, event scheduling, and strategies to handle late-arriving events.
事件可以随时随地发生,并且通常需要与其他制作者的事件进行协调。同步且一致的时间戳是跨分布式系统比较事件的硬性要求。
Events can happen anywhere and at any time and often need to be reconciled with events from other producers. Synchronized and consistent timestamps are a hard requirement for comparing events across distributed systems.
存储在事件流中的事件具有偏移量和时间戳。消费者使用偏移量来确定它已经读取了哪些事件,而指示事件创建时间的时间戳用于确定事件相对于其他事件发生的时间,并确保事件以正确的方式处理。命令。
An event stored in an event stream has both an offset and a timestamp. The offset is used by the consumer to determine which events it has already read, while the timestamp, which indicates when that event was created, is used to determine when an event occurred relative to other events and to ensure that events are processed in the correct order.
以下与时间戳相关的概念如图 6-1所示,它显示了它们在事件驱动工作流程中的时间位置:
The following timestamp-related concepts are illustrated in Figure 6-1, which shows their temporal positions in the event-driven workflow:
事件发生时生产者分配给事件的本地时间戳。
The local timestamp assigned to the event by the producer at the time the event occurred.
事件代理分配给事件的时间戳。您可以将其配置为事件时间或摄取时间,前者更为常见。在生产者的事件时间不可靠的情况下,代理摄取时间可以提供足够的替代。
The timestamp assigned to the event by the event broker. You can configure this to be either the event time or the ingestion time, with the former being much more common. In scenarios where the producer’s event time is unreliable, broker-ingestion time can provide a sufficient substitute.
消费者摄取事件的时间。这可以设置为代理记录中指定的事件时间,也可以是挂钟时间。
The time in which the event is ingested by the consumer. This can be set to the event time specified in the broker record, or it can be the wall-clock time.
消费者处理事件的挂钟时间。
The wall-clock time at which the event has been processed by the consumer.
您可以看到,可以通过事件代理将事件时间传播给使用者,使使用者逻辑能够根据事件发生的时间做出决策。这将有助于回答本章开头提出的三个问题。现在我们已经确定了时间戳的类型,让我们看看它们是如何生成的。
You can see that it’s possible to propagate the event time through the event broker to the consumer, enabling the consumer logic to make decisions based on when an event happened. This will help answer the three questions posed at the start of the chapter. Now that we’ve mapped out the types of timestamps, let’s take a look at how they’re generated.
物理学的一个基本限制是不能保证两个独立的系统具有完全相同的系统时钟时间。各种物理特性限制了系统时钟的精确度,例如底层时钟电路的材料容差、芯片工作温度的变化以及同步期间不一致的网络通信延迟。但是,可以建立几乎同步的本地系统时钟,并且最终足以满足大多数计算目的。
A fundamental limitation of physics is that two independent systems cannot be guaranteed to have precisely the same system-clock time. Various physical properties limit how precise system clocks can be, such as material tolerances in the underlying clock circuitry, variations in the operating temperature of the chip, and inconsistent network communication delays during synchronization. However, it is possible to establish local system clocks that are nearly in sync and end up being good enough for most computing purposes.
一致的时钟时间主要通过与网络时间协议 (NTP) 服务器同步来实现。亚马逊和谷歌等云服务提供商在各自地区提供冗余的卫星连接时钟和原子钟,以实现即时同步。
Consistent clock times are primarily accomplished by synchronizing with Network Time Protocol (NTP) servers. Cloud service providers such as Amazon and Google offer redundant satellite-connected and atomic clocks in their various regions for instant synchronization.
与局域网内的NTP服务器同步可以提供非常准确的本地系统时钟,15分钟后漂移仅几毫秒。根据 NTP 发明人 David Mills 的说法,在最佳情况下,通过更频繁的同步,可以将该时间减少到1 毫秒或更少,尽管间歇性网络问题可能会阻止在实践中达到该目标。开放互联网上的同步可能会导致更大的偏差,精度会降低到 +/- 100 毫秒的范围,如果您尝试重新同步来自全球不同地区的事件,这是一个需要考虑的因素。
Synchronization with NTP servers within a local area network can provide very accurate local system clocks, with a drift of only a few mS after 15 minutes. This can be reduced to 1 mS or less with more frequent synchronizations in best-case scenarios according to David Mills, NTP’s inventor, though intermittent network issues may prevent this target from being reached in practice. Synchronization across the open internet can result in much larger skews, with accuracy being reduced to ranges of +/− 100mS, and is a factor to be considered if you’re trying to resynchronize events from different areas of the globe.
NTP 同步也容易失败,因为网络中断、配置错误和瞬态问题可能会阻止实例同步。否则 NTP 服务器本身也可能变得不可靠或无响应。实例内的时钟可能会受到多租户问题的影响,就像共享底层硬件的基于 VM 的系统一样。
NTP synchronization is also prone to failure, as network outages, misconfiguration, and transient issues may prevent instances from synchronizing. The NTP servers themselves may also otherwise become unreliable or unresponsive. The clock within an instance may be affected by multitenancy issues, just as in VM-based systems sharing the underlying hardware.
对于绝大多数业务案例,频繁同步到 NTP 服务器可以为系统事件时间提供足够的一致性。NTP 服务器和 GPS 使用的改进已开始将 NTP 同步精度持续推向亚毫秒范围。分配为时间戳的创建时间和摄取时间值可以高度一致,但仍会出现轻微的乱序问题。本章稍后将介绍晚期事件的处理。
For the vast majority of business cases, frequent synchronization to NTP servers can provide sufficient consistency for system event time. Improvements to NTP servers and GPS usage have begun to push NTP synchronization accuracy consistently into the submillisecond range. The creation time and ingestion time values assigned as the timestamps can be highly consistent, though minor out-of-order issues will still occur. Handling of late events is covered later in this chapter.
时间戳提供了一种以一致的时间顺序处理分布在多个事件流和分区中的事件的方法。许多用例要求您根据时间维护事件之间的顺序,并且无论事件流何时处理,都需要一致、可重现的结果。使用偏移量作为比较方法仅适用于单个事件流分区内的事件,而事件通常需要从多个不同的事件流进行处理。
Timestamps provide a way to process events distributed across multiple event streams and partitions in a consistent temporal order. Many use cases require you to maintain order between events based on time, and need consistent, reproducible results regardless of when the event stream is processed. Using offsets as a means of comparison works only for events within a single event stream partition, while events quite commonly need to be processed from multiple different event streams.
银行必须确保存款和取款事件流都按照正确的时间顺序进行处理。它保留了取款和存款的状态运行记录,当客户的账户余额低于 0 美元时会施加透支罚款。对于本示例,银行将存款放在一个事件流中,将取款放在另一个事件流中,如图6-2所示。
A bank must ensure that both deposit and withdrawal event streams are processed in the correct temporal order. It keeps a stateful running tally of withdrawals and deposits, applying an overdraft penalty when a client’s account balance drops below $0. For this example, the bank has its deposits in one event stream and its withdrawals in another stream, as shown in Figure 6-2.
一种简单的消费和处理记录的方法(可能是循环处理器)可能会首先处理 10 美元的存款,然后处理 25 美元的取款(产生负余额和透支罚款),最后处理 20 美元的存款。然而,这是不正确的,并且并不代表事件发生的时间顺序。此示例清楚地表明,您在使用和处理事件时必须考虑事件的时间戳。下一节将更详细地讨论这一点。
A naive approach to consuming and processing records, perhaps a round-robin processor, might process the $10 deposit first, the $25 withdrawal second (incurring a negative balance and overdraft penalties), and the $20 deposit third. This is incorrect, however, and does not represent the temporal order in which the events occurred. This example makes clear that you must consider the event’s timestamp when consuming and processing events. The next section discusses this in greater detail.
确定性处理要求对事件进行一致的处理,以便可以在以后重现结果。事件调度是在从多个输入分区进行消费时选择下一个要处理的事件的过程。对于不可变的基于日志的事件流,记录按基于偏移的顺序消耗。然而,如图6-2所示,事件的处理顺序必须根据记录中提供的事件时间进行交错,无论它来自哪个输入分区,以确保结果正确。
Deterministic processing requires that events be processed consistently, such that the results can be reproduced at a later date. Event scheduling is the process of selecting the next events to process when consuming from multiple input partitions. For an immutable log-based event stream, records are consumed in an offset-based order. However, as Figure 6-2 demonstrates, the processing order of events must be interleaved based on the event time provided in the record, regardless of which input partition it comes from, to ensure correct results.
最常见的事件调度实现从所有分配的输入分区中选择具有最早时间戳的事件并将其分派到下游处理拓扑。
The most common event-scheduling implementation selects and dispatches the event with the oldest timestamp from all assigned input partitions to the downstream processing topology.
事件调度是许多流处理框架的一个功能,但在基本的消费者实现中通常不存在。您将需要确定您的微服务实施是否需要它。
Event scheduling is a feature of many stream-processing frameworks, but is typically absent from basic consumer implementations. You will need to determine if it is required for your microservice implementation.
如果事件的消费和处理顺序对业务逻辑很重要,那么您的微服务将需要事件调度。
Your microservice will need event scheduling if the order in which events are consumed and processed matters to the business logic.
一些流框架允许您实现自定义事件调度程序。例如,Apache Samza 允许您实现一个MessageChooser类,您可以在其中根据多种因素选择要处理的事件,例如某些事件流相对于其他事件流的优先级、挂钟时间、事件时间、事件元数据甚至内容在事件本身内。但是,在实现自己的事件调度程序时应该小心,因为许多自定义调度程序本质上是不确定的,如果需要重新处理,将无法生成可重现的结果。
Some streaming frameworks allow you to implement custom event schedulers. For example, Apache Samza lets you implement a MessageChooser class, where you select which event to process based on a number of factors, such as prioritization of certain event streams over others, the wall-clock time, event time, event metadata, and even content within the event itself. You should take care when implementing your own event scheduler, however, as many custom schedulers are nondeterministic in nature and won’t be able to generate reproducible results if reprocessing is required.
基于时间的事件处理顺序要求您选择用作事件时间戳的时间点,如图6-1所示。可以在本地分配的事件时间和代理摄取时间之间进行选择。两个时间戳在生产-消费工作流程中仅出现一次,而挂钟和消费者摄取时间根据应用程序的执行时间而变化。
A time-based order of event processing requires you to select which point in time to use as the event’s timestamp, as per Figure 6-1. The choice is between the locally assigned event time and broker ingestion time. Both timestamps occur only once each in a produce-consume workflow, whereas the wall-clock and consumer ingestion time change depending on when the application is executed.
在大多数情况下,特别是当所有消费者和所有生产者都健康并且任何消费者组都没有事件积压时,所有四个时间点将彼此相差几秒之内。相反,对于处理历史事件的微服务,事件时间和消费者摄取时间将显着不同。
In most scenarios, particularly when all consumers and all producers are healthy and there is no event backlog for any consumer group, all four points in time will be within a few seconds of each other. Contrarily, for a microservice processing historic events, event time and consumer ingestion time will differ significantly.
为了最准确地描述现实世界中的事件,最好使用本地分配的事件时间,前提是您可以信赖它的准确性。如果生产者的时间戳不可靠(并且您无法修复它),那么下一个最佳选择是根据事件被引入事件代理的时间来设置时间戳。仅在事件代理和生产者无法通信的极少数情况下,真实事件时间与代理分配的事件时间之间可能存在很大的延迟。
For the most accurate depiction events in the real world, it is best to use the locally assigned event time provided you can rely on its accuracy. If the producer has unreliable timestamps (and you can’t fix it), your next best bet is to set the timestamps based on when the events are ingested into the event broker. It is only in rare cases where the event broker and the producer cannot communicate that there may be a substantial delay between the true event time and the one assigned by the broker.
消费者必须知道记录的时间戳,然后才能决定如何排序进行处理。在消费者摄取时,时间戳提取器用于从消费事件中提取时间戳。该提取器可以从事件有效负载的任何部分获取信息,包括键、值和元数据。
The consumer must know the timestamp of the record before it can decide how to order it for processing. At consumer ingestion time, a timestamp extractor is used to extract the timestamp from the consumed event. This extractor can take information from any part of the event’s payload, including the key, value, and metadata.
每个消耗的记录都有一个由该提取器设置的指定事件时间时间戳。一旦设置了该时间戳,消费者框架就会在处理期间使用它。
Each consumed record has a designated event-time timestamp that is set by this extractor. Once this timestamp has been set, it is used by the consumer framework for the duration of its processing.
从事件驱动拓扑内向外部系统发出的任何非事件驱动请求都可能会引入不确定的结果。根据定义,外部系统是在微服务外部进行管理的,这意味着在任何时间点,它们的内部状态以及对请求微服务的响应都可能不同。这是否重要完全取决于微服务的业务需求,并由您评估。
Any non-event-driven requests made to external systems from within an event-driven topology may introduce nondeterministic results. By definition, external systems are managed externally to the microservice, meaning that at any point in time their internal state and their responses to the requesting microservice may differ. Whether this is significant depends entirely on the business requirements of your microservice and is up to you to assess.
水印用于通过处理拓扑来跟踪事件时间的进度,并声明给定事件时间(或更早)的所有数据已被处理。这是许多领先的流处理框架使用的常用技术,例如 Apache Spark、Apache Flink、Apache Samza 和 Apache Beam。谷歌的一份白皮书更详细地描述了水印,并为任何想要了解更多信息的人提供了一个很好的起点。
Watermarking is used to track the progress of event time through a processing topology and to declare that all data of a given event time (or earlier) has been processed. This is a common technique used by many of the leading stream-processing frameworks, such as Apache Spark, Apache Flink, Apache Samza, and Apache Beam. A whitepaper from Google describes watermarks in greater detail and provides a good starting point for anyone who would like to learn more about it.
水印是对同一处理拓扑中的下游节点的声明,即时间t及之前的所有事件都已被处理。然后,接收水印的节点可以更新其自己的内部事件时间并将其自己的水印向下游传播到其依赖的拓扑节点。该流程如图6-3所示。
A watermark is a declaration to downstream nodes within the same processing topology that all events of time t and prior have been processed. The node receiving the watermark can then update its own internal event time and propagate its own watermark downstream to its dependent topology nodes. This process is shown in Figure 6-3.
在此图中,消费者节点具有最高的水印时间,因为它从源事件流进行消费。周期性地生成新的水印,例如在经过一段挂钟或事件时间之后或者在处理了一些最小数量的事件之后。这些水印向下游传播到拓扑中的其他处理节点,这些节点相应地更新自己的事件时间。
In this figure, the consumer node has the highest watermark time because it’s consuming from the source event stream. New watermarks are generated periodically, such as after a period of wall-clock or event time has elapsed or after some minimum number of events has been processed. These watermarks propagate downstream to the other processing nodes in the topology, which update their own event time accordingly.
本章仅涉及水印,以便让您了解如何将它们用于确定性处理。如果您想更深入地研究水印,请考虑Tyler Akidau、Slava Chernyak 和 Reuven Lax 所著的优秀著作《流媒体系统》 (O'Reilly,2018 年)的第 2 章和第 3 章。
This chapter only touches on watermarks to give you an understanding of how they’re used for deterministic processing. If you would like to dig deeper into watermarks, consider Chapters 2 and 3 of the excellent book Streaming Systems, by Tyler Akidau, Slava Chernyak, and Reuven Lax (O’Reilly, 2018).
水印对于协调多个独立消费者实例之间的事件时间特别有用。图 6-4显示了两个消费者实例的简单处理拓扑。每个消费者实例使用来自其自己分配的分区的事件,应用一个groupByKey函数,然后应用一个aggregate函数。这需要一个shuffle,其中具有相同键的所有事件都被发送到单个下游聚合实例。在这种情况下,实例0和实例1的事件根据key相互发送,以确保相同key的所有事件都在同一个分区中。
Watermarks are particularly useful for coordinating event time between multiple independent consumer instances. Figure 6-4 shows a simple processing topology of two consumer instances. Each consumer instance consumes events from its own assigned partition, applies a groupByKey function, followed by an aggregate function. This requires a shuffle, where all events with the same key are sent to a single downstream aggregate instance. In this case, events from instance 0 and instance 1 are sent to each other based on the key to ensure all events of the same key are in the same partition.
该图中有相当多的内容需要解开,所以让我们从头开始看一下。
There is a fair bit to unpack in this diagram, so let’s take a look at it from the start.
水印是在源函数处生成的,其中事件是从事件流分区消耗的。水印定义了该消费者的事件时间,并随着消费者节点的事件时间的增加而向下游传播(图 6-4中的#1 )。
Watermarks are generated at the source function, where the events are consumed from the event stream partition. The watermarks define the event time at that consumer and are propagated downstream as the event time of the consumer node is incremented (#1 in Figure 6-4).
当水印到达时,下游节点更新其事件时间,并依次生成自己的新水印以向下游传播到其后继节点。具有多个输入的节点(例如 )aggregate消耗来自多个上游输入的事件和水印。节点的事件时间是节点内部跟踪的所有输入源事件时间的最小值(图 6-4中的#2 )。
Downstream nodes update their event time as the watermarks arrive, and in turn generate their own new watermark to propagate downstream to its successors. Nodes with multiple inputs, such as aggregate, consume events and watermarks from multiple upstream inputs. The node’s event time is the minimum of all of its input sources’ event times, which the node keeps track of internally (#2 in Figure 6-4).
在示例中,aggregate一旦来自 groupByKey-1 节点的水印到达(图6-4中的#3 ),两个节点的事件时间都会从 13 更新到 15。注意,水印不影响节点的事件调度;它只是通知节点它应该将时间戳早于水印的任何事件视为延迟。本章稍后将介绍处理延迟事件。
In the example, both aggregate nodes will have their event time updated from 13 to 15 once the watermark from the groupByKey-1 node arrives (#3 in Figure 6-4). Note that the watermark does not affect the event scheduling of the node; it simply notifies the node that it should consider any events with a timestamp earlier than the watermark to be considered late. Handling late events is covered later in this chapter.
Spark、Flink 和 Beam 等重量级处理框架需要专用的处理资源集群来大规模执行流处理。这是特别相关的,因为该集群还提供了跨任务通信和每个处理任务的集中协调的手段。重新分区事件(例如本示例中的groupByKey+aggregate操作)使用集群内部通信,而不是事件代理中的事件流。
Spark, Flink, and Beam, among other heavyweight processing frameworks, require a dedicated cluster of processing resources to perform stream processing at scale. This is particularly relevant because this cluster also provides the means for cross-task communications and centralized coordination of each processing task. Repartitioning events, such as with the groupByKey + aggregate operation in this example, use cluster-internal communications and not event streams in the event broker.
在流处理器中维护时间的第二种选择(简称为流时间)是 Apache Kafka Streams 所青睐的方法。从一个或多个事件流读取的消费者应用程序维护其拓扑的流时间,这是已处理事件的最高时间戳。使用者实例使用并缓冲分配给它的每个事件流分区中的事件,应用事件调度算法来选择下一个要处理的事件,然后如果流时间大于前一个流时间,则更新流时间。流媒体时间永远不会减少。
A second option for maintaining time in a stream processor, known simply as stream time, is the approach favored by Apache Kafka Streams. A consumer application reading from one or more event streams maintains a stream time for its topology, which is the highest timestamp of processed events. The consumer instance consumes and buffers events from each event stream partition assigned to it, applies the event-scheduling algorithm to select the next event to process, and then updates the stream time if it is larger than the previous stream time. Stream time will never be decreased.
图 6-5显示了流时间的示例。消费者节点根据其收到的最高事件时间值维护单个流时间。流时间当前设置为 20,因为这是最近处理的事件的事件时间。下一个要处理的事件是两个输入缓冲区中的最小值 - 在本例中,它是事件时间为 30 的事件。该事件将向下分派到处理拓扑,并且流时间将更新为 30。
Figure 6-5 shows an example of stream time. The consumer node maintains a single stream time based on the highest event-time value it has received. The stream time is currently set to 20 since that was the event time of the most recently processed event. The next event to be processed is the smallest value of the two input buffers—in this case, it’s the event with event time 30. The event is dispatched down to the processing topology, and the stream time will be updated to 30.
通过在处理下一个事件之前完全通过拓扑处理每个事件来维护流时间。在拓扑包含重新分区流的情况下,每个拓扑被分成两个,并且每个子拓扑维护其自己的不同流时间。事件以深度优先的方式处理,以便在任何给定时间在子拓扑中仅处理一个事件。这与基于水印的方法不同,在基于水印的方法中,事件可以在每个处理节点的输入处缓冲,并且每个节点的事件时间独立更新。
Stream time is maintained by processing each event completely through the topology before processing the next one. In cases where a topology contains a repartition stream, each topology is split into two, and each subtopology maintains its own distinct stream time. Events are processed in a depth-first manner, such that only one event is being processed in a subtopology at any given time. This is different than the watermark-based approach where events can be buffered at the inputs of each processing node, with each node’s event time independently updated.
再次考虑图 6-4中相同的两个实例消费者示例,但这次使用 Kafka Streams 倡导的流时间方法(参见图 6-6)。一个显着的区别是 Kafka Streams 方法使用所谓的内部事件流将重新分区的事件发送回事件代理。然后,实例重新使用该流,所有重新分区的数据按键共同位于单个分区内。这在功能上与重量级集群内的 shuffle 机制相同,但不需要专用集群(注意:Kafka Streams 对微服务非常友好)。
Consider again the same two-instance consumer example from Figure 6-4, but this time with the stream time approach championed by Kafka Streams (see Figure 6-6). A notable difference is that the Kafka Streams approach sends the repartitioned events back to the event broker using what’s known as an internal event stream. This stream is then reconsumed by the instances, with all repartitioned data colocated by key within single partitions. This is functionally the same as the shuffle mechanism within the heavyweight cluster, but does not require a dedicated cluster (note: Kafka Streams is very microservice friendly).
在此示例中,输入流中的事件根据其键重新分区,并写入重新分区事件流中。事件以 为关键A并B以 结束P1,而事件以 为关键X并Z以 结束P0。另请注意,每个事件的事件时间均已维护,并且不会被当前挂钟时间覆盖。回想一下,重新分区只能被视为现有事件数据的逻辑混洗。重写事件的事件时间会完全破坏原来的时间顺序。
In this example, events from the input stream are repartitioned according to their key and are written into the repartition event stream. The events keyed on A and B end up in P1, while the events keyed on X and Z end up in P0. Also note that the event time has been maintained for each event, and not overwritten by the current wall-clock time. Recall that the repartition should be treated only as a logical shuffling of existing event data. Rewriting the event time of an event would completely destroy the original temporal ordering.
请注意图中所示的子拓扑。由于重新分区事件流,处理拓扑实际上被减半,这意味着每个子拓扑上的工作可以并行完成。子拓扑1和3消费重新分区流并将事件组合在一起,同时子拓扑0和2生成重新分区事件。每个子拓扑都维护自己的流时间,因为两者都消耗独立的事件流。
Notice the subtopologies shown in the figure. Because of the repartition event stream, the processing topology is effectively cut in half, meaning that work on each subtopology can be done in parallel. Subtopologies 1 and 3 consume from the repartition stream and group events together, while subtopologies 0 and 2 produce the repartitioned events. Each subtopology maintains its own stream time, since both are consuming from independent event streams.
在理想的情况下,所有事件都可以毫无问题地生成,并以零延迟的方式提供给消费者。不幸的是,对于生活在现实世界中的我们所有人来说,情况并非如此,因此我们必须计划适应无序事件。如果事件的时间戳不等于或大于事件流中位于其前面的事件,则称该事件是无序的。在图6-7中,事件F是无序的,因为它的时间戳小于G's,就像事件H是无序的,因为它的时间戳小于I's。
In an ideal world, all events are produced without issue and available to the consumer with zero latency. Unfortunately for all of us living in the real world, this is never the case, so we must plan to accommodate out-of-order events. An event is said to be out of order if its timestamp isn’t equal to or greater than the events ahead of it in the event stream. In Figure 6-7, event F is out of order because its timestamp is lower than G’s, just as event H is out of order as its timestamp is lower than I’s.
有界数据集(例如批量处理的历史数据)通常对无序数据具有相当的弹性。整个批次可以被认为是一个大窗口,并且如果该批次的处理尚未开始,则无序到达几分钟甚至几小时的事件并不真正相关。通过这种方式,批量处理的有界数据集可以产生具有高确定性的结果。这是以高延迟为代价的,特别是对于传统的夜间批量大数据处理作业,只有在 24 小时加上批处理时间后才能获得结果。
Bounded data sets, such as historical data processed in batch, are typically fairly resilient to out-of-order data. The entire batch can be thought of as one large window, and an event arriving out of order by many minutes or even hours is not really relevant provided that the processing for that batch has not yet started. In this way, a bounded data set processed in batch can produce results with high determinism. This comes at the expense of high latency, especially for the traditional sorts of nightly batch big-data processing jobs where the results are available only after the 24-hour period, plus batch processing time.
对于无限的数据集,例如不断更新的事件流中的数据集,开发人员在设计微服务时必须考虑延迟和确定性的要求。这超出了技术要求,延伸到了业务需求,因此任何事件驱动的微服务开发人员都必须问:“我的微服务是否根据业务要求处理无序和迟到事件?” 无序事件要求企业就如何处理它们做出具体决策,并确定延迟优先还是确定性优先。
For unbounded data sets, such as those in ever-updating event streams, the developer must consider the requirements of latency and determinism when designing the microservice. This extends beyond the technological requirements into the business requirements, so any event-driven microservice developer must ask, “Does my microservice handle out-of-order and late-arriving events according to business requirements?” Out-of-order events require the business to make specific decisions about how to handle them, and to determine whether latency or determinism takes priority.
考虑前面的银行帐户示例。存款后立即提款必须按正确的顺序进行处理,以免错误地收取透支费用,无论活动的顺序如何或延迟多长时间。为了缓解这种情况,应用程序逻辑可能需要在业务指定的时间段(例如一小时的宽限窗口)内维护状态以处理无序数据。
Consider the previous example of the bank account. A deposit followed by an immediate withdrawal must be processed in the correct order lest an overdraft charge be incorrectly applied, regardless of the ordering of events or how late they may be. To mitigate this, the application logic may need to maintain state to handle out-of-order data for a time period specified by the business, such as a one-hour grace window.
来自单个分区的事件应始终根据其偏移顺序进行处理,无论其时间戳如何。这可能会导致无序事件。
Events from a single partition should always be processed according to their offset order, regardless of their timestamp. This can lead to out-of-order events.
仅当从消费微服务的角度来看时,事件才可以被视为延迟。一个微服务可能会将任何无序事件视为迟到,而另一个微服务可能相当宽容,并且需要经过许多小时的挂钟或事件时间才能认为事件迟到。
An event can be considered late only when viewed from the perspective of the consuming microservice. One microservice may consider any out-of-order events as late, whereas another may be fairly tolerant and require many hours of wall-clock or event time to pass before considering an event to be late.
考虑两个事件,一个事件的时间为t,另一个事件的时间为t '。事件t ′ 的时间戳早于事件t。
Consider two events, one with time t, the other with time t′. Event t′ has an earlier timestamp than event t.
当事件t ' 在水印W(t)之后到达时,它被认为是迟到的。如何处理这个事件取决于具体节点。
The event t′ is considered late when it arrives after the watermark W(t). It is up to the specific node how to handle this event.
当事件t ' 在流时间增加超过t '之后到达时,事件 t ' 被视为迟到。如何处理该事件取决于子拓扑中的每个操作员。
The event t′ is considered late when it arrives after the stream time has been incremented past t′. It is up to each operator in the subtopology how to handle this event.
仅当事件错过了特定于消费者的截止日期时,事件才会延迟。
An event is late only when it has missed a deadline specific to the consumer.
There are several ways that out-of-order events can occur.
当然,最明显的是事件源自无序数据。当从已经无序的流中消耗数据或从具有现有无序时间戳的外部系统获取事件时,可能会发生这种 情况。
The most obvious, of course, is when events are sourced from out-of-order data. This can occur when data is consumed from a stream that is already out of order or when events are being sourced from an external system with existing out-of-order timestamps.
多个生产者写入多个输出分区可能会引入乱序事件。重新分区现有事件流是发生这种情况的一种方法。图6-8显示了两个消费者实例对两个分区的重新分区。在此场景中,源事件指示用户与哪个产品进行了交互。例如,Harry 与产品 ID12 和 ID77 进行了交互。假设数据分析师需要在用户 ID 上重新设置这些事件的密钥,以便他们可以对用户的参与度执行基于会话的分析。由此产生的输出流可能会出现一些无序事件。
Multiple producers writing to multiple output partitions can introduce out-of-order events. Repartitioning an existing event stream is one way in which this can happen. Figure 6-8 shows the repartitioning of two partitions by two consumer instances. In this scenario the source events indicate which product the user has interacted with. For instance, Harry has interacted with products ID12 and ID77. Say that a data analyst needs to rekey these events on the user ID, such that they can perform session-based analysis of the user’s engagements. The resultant output streams may end up with some out-of-order events.
请注意,每个实例都维护自己的内部流时间,并且两个实例之间没有同步。这可能会导致时间偏差,从而产生无序事件,如图6-9所示。
Note that each instance maintains its own internal stream time and that there is no synchronization between the two instances. This can cause a time skew that produces out-of-order events, as shown in Figure 6-9.
实例 0 在流时间上仅稍微领先于实例 1,但由于它们的流时间独立,时间t = 90 和t = 95的事件在重新分区的事件流中被视为无序。不平衡的分区大小、不平等的处理速率和大量事件积压会加剧此问题。这里的影响是,以前按顺序的事件数据现在是无序的,因此作为消费者,您不能依赖于每个事件流中持续增加的时间。
Instance 0 was only slightly ahead of instance 1 in stream time, but because of their independent stream times, the events of time t = 90 and t = 95 are considered out of order in the repartitioned event stream. This issue is exacerbated by unbalanced partition sizes, unequal processing rates, and large backlogs of events. The impact here is that the previously in-order event data is now out of order, and thus as a consumer you cannot depend on having consistently incrementing time in each of your event streams.
单线程生产者在正常操作中不会创建乱序事件,除非它从乱序源获取数据。
A single-threaded producer will not create out-of-order events in normal operation unless it is sourcing its data from an out-of-order source.
由于只要检测到具有较高时间戳的事件,流时间就会增加,因此可能会出现大量事件由于重新排序而被视为延迟的情况。这可能会对处理产生影响,具体取决于消费者选择如何处理无序事件。
Since the stream time is incremented whenever an event with a higher timestamp is detected, it is possible to end up in a scenario where a large number of events are considered late due to reordering. This may have an effect on processing depending on how the consumers choose to handle out-of-order events.
延迟事件主要关注基于时间的业务逻辑,例如聚合特定时间段内的事件或在经过特定时间段后触发事件。延迟事件是在业务逻辑已经完成该特定时间段的处理之后到达的事件。窗口函数是基于时间的业务逻辑的一个很好的例子。
Late events are predominantly the concern of time-based business logic, such as aggregating events in a particular time period or triggering an event after a certain period of time has passed. A late event is one that arrives after the business logic has already finished processing for that particular period of time. Windowing functions are an excellent example of time-based business logic.
窗口化意味着按时间将事件分组在一起。这对于具有相同键的事件特别有用,您希望了解该键的事件在该时间段内发生了什么。事件窗口分为三种主要类型,但再次强调,请务必检查您的流处理框架以获取更多信息。
Windowing means grouping events together by time. This is particularly useful for events with the same key, where you want to see what happened with events of that key in that period of time. There are three main types of event windows, but again, be sure to check your stream-processing framework for more information.
窗口化可以使用事件时间或处理时间来完成,尽管事件时间窗口化通常有更多的业务应用程序。
Windowing can be done using either event time or processing time, though event-time windowing typically has more business applications.
翻滚窗口是固定大小的窗口。前一个和后一个窗口不重叠。图 6-10显示了三个滚动窗口,每个窗口都在t、t + 1 等上对齐。这种窗口可以帮助回答诸如“产品使用高峰时间是什么时候?”之类的问题。
A tumbling window is a window of a fixed size. Previous and subsequent windows do not overlap. Figure 6-10 shows three tumbling windows, each aligned on t, t + 1, and so on. This sort of windowing can help answer questions such as “When is the peak hour for product usage?”
滑动窗口具有固定的窗口大小和增量步长,称为窗口滑动。它必须仅反映窗口中当前事件的聚合。滑动窗口可以帮助回答诸如“过去一小时内有多少用户点击了我的产品?”之类的问题。图6-11显示了滑动窗口的示例,包括窗口的大小和向前滑动的量。
A sliding window has a fixed window size and incremental step known as the window slide. It must reflect only the aggregation of events currently in the window. A sliding window can help answer questions such as “How many users clicked on my product in the past hour?” Figure 6-11 shows an example of the sliding window, including the size of the window and the amount that it slides forward.
会话窗口是动态调整大小的窗口。它根据由于不活动而超时而终止,并为超时后发生的任何活动启动新会话。图 6-12显示了会话窗口的示例,其中由于用户 C 不活动而出现会话间隙。这种窗口可以帮助回答诸如“用户在给定浏览会话中看什么?”之类的问题。
A session window is a dynamically sized window. It is terminated based on a timeout due to inactivity, with a new session started for any activity happening after the timeout. Figure 6-12 shows an example of session windows, with a session gap due to inactivity for user C. This sort of window can help answer questions such as “What does a user look at in a given browsing session?”
每个窗口函数都必须处理乱序事件。您必须决定等待任何无序事件多久,以免认为它们为时已晚而无法考虑。流处理中的基本问题之一是您永远无法确定已收到所有事件。等待无序事件可以起作用,但最终您的服务将需要放弃,因为它不能无限期地等待。其他需要考虑的因素包括要存储多少状态、延迟事件的可能性以及不使用延迟事件的业务影响。
Each of these window functions must deal with out-of-order events. You must decide how long to wait for any out-of-order events before deeming them too late for consideration. One of the fundamental issues in stream processing is that you can never be sure that you have received all of the events. Waiting for out-of-order events can work, but eventually your service will need to give up, as it cannot wait indefinitely. Other factors to consider include how much state to store, the likelihood of late events, and the business impact of not using the late events.
在开发工程解决方案之前,应在业务级别确定处理无序和迟到事件的策略,因为策略会根据数据的重要性而变化。可能需要处理诸如金融交易和系统故障之类的关键事件,无论它们在流中的位置如何。或者,测量类型的事件(例如温度或力度量)可能因不再相关而被简单地丢弃。
The strategy for handling out-of-order and late-arriving events should be determined at a business level prior to developing an engineering solution, as strategies will vary depending on the importance of the data. Critical events such as financial transactions and system failures may be required to be handled regardless of their position in the stream. Alternately, measurement-style events, such as temperature or force metrics, may simply be discarded as no longer relevant.
业务需求还决定了可接受的延迟时间,因为等待事件到达可能会增加确定性,但代价是更高的延迟。这可能会对时间敏感的应用程序或具有严格服务级别协议的应用程序的性能特征产生负面影响。值得庆幸的是,微服务提供了必要的灵活性,可以根据每个服务定制确定性、延迟和无序事件处理特征。
Business requirements also dictate how much latency is acceptable, as waiting for events to arrive may increase determinism but at the expense of higher latency. This can negatively affect the performance characteristics of time-sensitive applications or those with tight service-level agreements. Thankfully, microservices provide the flexibility necessary to tailor determinism, latency, and out-of-order event handling characterstics on a per-service basis.
无论您的框架是否使用水印或流时间,都可以通过多种方式处理迟到事件。
There are several ways in which a late-arriving event can be handled, regardless of whether your framework uses watermarks or stream time.
只需删除该事件即可。窗口已关闭,任何基于时间的聚合都已完成。
Simply drop the event. The window is closed, and any time-based aggregations are already complete.
延迟窗口结果的输出,直到经过固定的时间。这会带来更高的确定性,但代价是增加延迟。旧窗口需要保持可用于更新,直到经过预定的时间。
Delay output of the window results until a fixed amount of time has passed. This incurs higher determinism at the expense of increased latency. The old window(s) need to remain available for updating until the predetermined amount of time has passed.
一旦窗口被认为完成,就输出窗口结果。然后,在预定的宽限期内保持窗口可用。每当该窗口的延迟事件到达时,就更新聚合并输出新更新的聚合。这与等待策略类似,只不过更新是在延迟事件到达时生成的。
Output the windowed result as soon as the window is deemed complete. Then, keep the window(s) around and available for the predetermined grace period. Whenever a late event arrives for that window, update the aggregation and output the newly updated aggregation. This is similar to the wait strategy, except updates are generated as late events arrive.
无论微服务等待多长时间,事件最终都会为时已晚,需要被丢弃。对于微服务应如何处理延迟事件,没有固定的技术规则;只需确保您的业务需求得到充分满足即可。如果微服务的业务需求中未指定处理延迟事件的协议,则业务必须努力解决该问题。
Regardless of how long a microservice waits, eventually events will simply be too late and will need to be discarded. There is no cut-and-dry technical rule for how your microservice should handle late events; just ensure that your business requirements are sufficiently met. If the protocol for handling late events is not specified in the microservice’s business requirements, then the business must work to resolve that.
以下是一些问题,可帮助您确定处理延迟事件的良好准则:
Here are a few questions to help you determine good guidelines for handling late events:
迟到事件发生的可能性有多大?
How likely are late events to occur?
您的服务需要多长时间来防范延迟事件?
How long does your service need to guard against late events?
放弃延迟事件会对业务产生哪些影响?
What are the business impacts of dropping late events?
等待很长时间才能捕获迟到的事件有哪些商业好处?
What are the business benefits of waiting a long time to capture late events?
需要多少磁盘或内存来维护状态?
How much disk or memory does it take to maintain state?
等待迟到事件所产生的费用是否超过其收益?
Do the expenses incurred in waiting for late events outweigh the benefits?
不可变事件流提供了倒回消费者组偏移并从任意时间点重播处理的能力。这称为重新处理,这是每个事件驱动的微服务在设计中都需要考虑的问题。重新处理通常仅在使用事件时间处理事件的事件驱动微服务上执行,而不是在那些依赖于挂钟时间聚合和窗口的微服务上执行。
Immutable event streams provide the ability to rewind consumer group offsets and replay processing from an arbitrary point in time. This is known as reprocessing, and it’s something that every event-driven microservice needs to consider in its design. Reprocessing is typically performed only on event-driven microservices that use event time for processing events, and not those that rely on wall-clock time aggregations and windowing.
事件调度是能够正确地重新处理事件流中的历史数据的重要组成部分。它确保微服务按照近乎实时的相同顺序处理事件。处理乱序事件也是此过程的重要组成部分,因为通过事件代理重新分区事件流(而不是使用 Spark、Flink 或 Beam 等重量级框架)可能会导致乱序事件。
Event scheduling is an important part of being able to correctly reprocess historical data from event streams. It ensures that microservices process events in the same order that they did during near–real time. Handling out-of-order events is also an important part of this process, as repartitioning an event stream through the event broker (instead of using a heavyweight framework like Spark, Flink, or Beam) can cause out-of-order events.
当您想要重新处理事件流时,请遵循以下步骤:
Here are the steps to follow when you want to reprocess your event streams:
确定起点。作为最佳实践,所有有状态使用者都应该从他们订阅的每个事件流的一开始就重新处理事件。这尤其适用于实体事件流,因为它们包含有关所讨论实体的重要事实。
Determine the starting point. As a best practice, all stateful consumers should reprocess events from the very beginning of each event stream that they are subscribed to. This applies to entity event streams in particular, as they contain important facts about the entities under question.
确定要重置哪些消费者偏移量。任何包含状态处理中使用的事件的流都应该重置到最开始的位置,因为如果您从错误的位置开始,则很难确保最终得到正确的状态(考虑一下如果您重新处理某人的银行余额会发生什么)并且不小心遗漏了以前的薪水)。
Determine which consumer offsets to reset. Any streams that contain events used in stateful processing should be reset to the very beginning, as it is difficult to ensure that you will end up with a correct state if you start in the wrong location (consider what would happen if you reprocessed someone’s bank balance and accidentally omitted previous paychecks).
考虑数据量。一些微服务可能会处理大量事件。考虑重新处理事件可能需要多长时间以及可能存在的任何瓶颈。可能需要配额(请参阅“配额”)来确保您的微服务不会因 I/O 而导致事件代理不堪重负。此外,如果您希望生成大量经过重新处理的输出数据,您可能需要通知任何下游消费者。如果未启用自动扩展(请参阅“自动扩展应用程序” ) ,他们可能需要相应地扩展其服务。
Consider the volume of data. Some microservices may process huge quantities of events. Consider how long it may take to reprocess the events, and any bottlenecks that may exist. Quotas (see “Quotas”) may be needed to ensure that your microservice doesn’t overwhelm the event broker with I/O. Additionally, you may need to notify any downstream consumers if you are expecting to generate large amounts of reprocessed output data. They may need to scale their services accordingly if autoscaling (see “Autoscaling Applications”) is not enabled.
考虑重新处理的时间。重新处理可能需要几个小时才能完成,因此值得计算您可能需要多少停机时间。确保您的下游消费者在服务重新处理时也能接受可能陈旧的数据。将消费者实例的数量扩展到最大并行度可以显着减少停机时间,并且可以在重新处理完成后缩小停机时间。
Consider the time to reprocess. It is possible that reprocessing may take many hours to do, so it’s worth calculating how much downtime you may need. Ensure that your downstream consumers are also okay with possibly stale data while your service reprocesses. Scaling the number of consumer instances to maximum parallelism can significantly reduce this downtime and can be scaled down once reprocessing has completed.
考虑影响。某些微服务执行重新处理时您可能不希望发生的操作。例如,在包裹发货后向用户发送电子邮件的服务不应在重新处理事件时重新向用户发送电子邮件,因为这将是一种糟糕的用户体验,并且从业务角度来看完全没有意义。仔细考虑再处理对系统业务逻辑的影响,以及下游 消费者可能出现的潜在问题。
Consider the impact. Some microservices perform actions that you may not want to occur when reprocessing. For instance, a service that emails users when their packages have shipped should not re-email users when reprocessing events, as it would be a terrible user experience and completely nonsensical from a business perspective. Carefully consider the impact of reprocessing on the business logic of your system, as well as potential issues that may arise for downstream consumers.
事件在近实时处理期间可能会延迟(水印或流时间增加),但在事件流重新处理期间可能会按预期在事件流中可用。这个问题可能很难检测到,但它确实说明了事件驱动的微服务的互联本质以及上游问题如何影响下游消费者。让我们快速了解一下这是如何发生的。
An event may be late during near-real-time processing (watermark or stream time is incremented) but may be available as expected within the event stream during event-stream reprocessing. This issue can be hard to detect, but it really illustrates the connected nature of event-driven microservices and how upstream problems can affect downstream consumers. Let’s take a quick look at how this can occur.
在这种情况下,记录是按时间戳顺序创建的,但要稍后才能发布(参见图 6-13)。在正常操作期间,生产者在事件发生时发送事件,而消费者则近乎实时地消费它们。这种情况很难确定何时发生,即使回想起来也可能被忽视。
In this scenario, records are created in timestamp order but can’t be published until a later time (see Figure 6-13). During normal operation, producers send their events as they occur, and consumers consume them in near-real-time. This scenario is tricky to identify when it’s happening and can go unnoticed even in retrospect.
假设生产者有许多记录准备发送,但无法连接到事件代理。这些记录带有事件发生的当地时间的时间戳。生产者将重试多次,最终要么成功,要么放弃并失败(理想情况下是有噪音的失败,以便可以识别并纠正错误的连接)。该场景如图6-14所示。来自流 A 的事件仍会被消耗,水印/流时间也会相应增加。然而,在从流 B 消费后,消费者最终没有新事件,因此可以简单地假设没有新数据可用。
Say a producer has a number of records ready to send, but it is unable to connect to the event broker. The records are timestamped with the local time that the event occurred. The producer will retry a number of times and either eventually succeed or give up and fail (ideally a noisy failure so that the faulty connection can be identified and rectified). This scenario is shown in Figure 6-14. Events from stream A are still consumed, with the watermark/stream time incremented accordingly. However, upon consuming from stream B, the consumer ends up with no new events, so it can simply assume that no new data is available.
最终,生产者将能够将记录写入事件流。这些事件按照实际发生的正确事件时间顺序发布,但由于挂钟延迟,近实时消费者会将它们标记为迟到并如此对待它们。如图 6-15所示。
Eventually the producer will be able to write records to the event stream. These events are published in the correct event-time order that they actually occurred, but because of the wall-clock delay, near-real-time consumers will have marked them as late and treat them as such. This is shown in Figure 6-15.
缓解这种情况的一种方法是在处理事件之前等待预定的时间,尽管这种方法确实会产生延迟成本,并且仅在生产延迟短于等待时间时才有用。另一种选择是在代码中使用强大的后期事件处理逻辑,这样您的业务逻辑就不会受到这种情况的影响。
One way to mitigate this is to wait a predetermined amount of time before processing events, though this approach does incur a latency cost and will only be useful when production delays are shorter than the wait time. Another option is to use robust late-event-handling logic in your code such that your business logic is not impacted by this scenario.
本章着眼于决定论以及如何最好地利用无界流来处理它。它还研究了如何在多个分区之间选择要处理的下一个事件,以确保在近实时处理和重新处理时尽最大努力确定性。无限的事件流与间歇性故障相结合的本质意味着完全的确定性永远无法完全实现。大部分时间都有效的合理、尽力而为的解决方案可以在延迟和正确性之间提供最佳权衡。
This chapter looked at determinism and how best to approach it with unbounded streams. It also examined how to select the next events to process between multiple partitions to ensure best-effort determinism when processing in both near-real-time and when reprocessing. The very nature of an unbounded stream of events combined with intermittent failures means that full determinism can never be completely achieved. Reasonable, best-effort solutions that work most of the time provide the best tradeoff between latency and correctness.
无序和迟到事件是您在设计中必须考虑的因素。本章探讨了如何使用水印和流时间来识别和处理这些事件。如果您想了解有关水印的更多信息,请查看 Tyler Akidau 的优秀文章“批流处理之外的世界 101”和102。关于分布式系统时间的其他考虑因素和见解可以在 Mikito Takada 的在线书籍《Distributed Systems for Fun and Profit》中找到。
Out-of-order and late-arriving events are factors that you must consider in your designs. This chapter explored how watermarks and stream time can be used to identify and handle these events. If you would like to read more about watermarks, check out Tyler Akidau’s excellent articles, “The World Beyond Batch Streaming 101” and 102. Additional considerations and insights into distributed system time can be found in Mikito Takada’s online book Distributed Systems for Fun and Profit.
有状态流是事件驱动微服务最重要组件的基础,因为大多数应用程序需要维护某种程度的状态来满足其处理要求。“从实体事件具体化状态”简要介绍了将事件流具体化为本地状态的原则。本章更深入地了解如何构建、管理和使用事件驱动的微服务的状态。
Stateful streaming underpins the most important components of an event-driven microservice, as most applications will need to maintain some degree of state for their processing requirements. “Materializing State from Entity Events” briefly covered the principles of materializing an event stream into local state. This chapter takes a much deeper look at how to build, manage, and use state for event-driven microservices.
Let’s start with some definitions:
来自源事件流的事件投影(不可变)
A projection of events from the source event stream (immutable)
服务的业务状态存储位置(可变)
Where your service’s business state is stored (mutable)
有状态微服务中需要并广泛使用物化状态和状态存储,但区分它们很重要。物化状态使您能够在微服务应用程序中使用常见的业务实体,而状态存储使您能够存储业务状态和中间 计算。
Both materialized state and state stores are required and used extensively in stateful microservices, but it’s important to distinguish between them. Materialized states enable you to use common business entities in your microservice applications, whereas state stores enable you to store business state and intermediate computations.
每个微服务设计还必须考虑服务将存储其数据的位置。存储和访问状态有两个主要选项:
Each microservice design must also take into account where the service will store its data. There are two main options for storing and accessing state:
Internally, such that the data is stored in the same container as the processor, allocated in memory or on disk.
Externally, such that the data is stored outside of the processor’s container, in some form of external storage service. This is often done across a network.
图 7-1显示了内部和外部状态存储的示例。
Figure 7-1 shows examples of both internal and external state storage.
选择内部或外部存储主要取决于微服务的业务职责和技术要求。但在更深入地评估这两个选项之前,您需要考虑变更日志的作用。
The choice of internal or external storage depends primarily on the microservice’s business responsibilities and technical requirements. But before evaluating these two options in greater depth, you’ll need to consider the role of the changelog.
更改日志是对状态存储数据所做的所有更改的记录。它是表流二元性中的流,状态表转换为单个事件的流。作为微服务实例外部维护的状态的永久副本,更改日志可用于重建状态,如图7-2所示,并作为检查点事件处理进度的一种方式。
A changelog is a record of all changes made to the data of the state store. It is the stream in the table-stream duality, with the table of state transformed into a stream of individual events. As a permanent copy of the state maintained outside of the microservice instance, the changelog can be used to rebuild state, as shown in Figure 7-2, and serve as a way of checkpointing event processing progress.
更改日志优化了重建失败服务的任务,因为它们存储了先前处理的结果,允许恢复处理器避免重新处理所有输入事件。
Changelogs optimize the task of rebuilding failed services because they store the results of previous processing, allowing a recovering processor to avoid reprocessing all input events.
变更日志流与任何其他流一样存储在事件代理中,并且如前所述,提供了重建状态存储的方法。变更日志流应该被压缩,因为它们只需要最新的键/值对来重建状态。
Changelog streams are stored in the event broker just like any other stream and, as noted, provide a means of rebuilding the state store. Changelog streams should be compacted since they only need the most recent key/value pair to rebuild the state.
更改日志可以以高性能的方式扩展和恢复状态,特别是对于内部状态存储。在这两种情况下,新创建的应用程序实例只需要从关联的变更日志分区加载数据,如图7-3所示。
Changelogs can scale and recover state in a highly performant manner, especially for internal state stores. In both cases, the newly created application instance just needs to load the data from the associated changelog partitions, as shown in Figure 7-3.
变更日志要么作为内置功能提供(例如在Kafka Streams 客户端中),要么由应用程序开发人员实现。基本生产者/消费者客户端往往不提供任何变更日志或状态支持。
Changelogs are either provided as a built-in feature, such as in the Kafka Streams client, or implemented by the application developer. Basic producer/consumer clients tend not to provide any changelogging or stateful support.
内部状态存储与微服务的业务逻辑共存于同一容器或虚拟机环境中。具体来说,内部状态存储的存在与微服务实例的存在相关,两者都运行在相同的底层硬件上。
Internal state stores coexist in the same container or VM environment as the microservice’s business logic. Specifically, the existence of the internal state store is tied to the existence of the microservice instance, with both running on the same underlying hardware.
每个微服务实例都会具体化来自其分配分区的事件,从而使每个分区的数据在存储中逻辑上保持独立。这些逻辑上独立的物化分区允许微服务实例在消费者组重新平衡后简单地删除已撤销分区的状态。通过确保物化状态仅存在于拥有分区的实例上,可以避免资源泄漏和多个事实来源。可以通过使用事件流或更改日志中的输入事件来重建新的分区分配。
Each microservice instance materializes the events from its assigned partitions, keeping each partition’s data logically separate within the store. These logically separate materialized partitions permit a microservice instance to simply drop the state for a revoked partition after a consumer group rebalance. This avoids resource leaks and multiple sources of truth by ensuring that materialized state exists only on the instance that owns the partition. New partition assignments can be rebuilt by consuming the input events from the event stream or from the changelog.
高性能键/值存储(例如 RocksDB)通常用于实现内部状态存储,并经过优化以提高本地固态驱动器 (SSD) 的效率,从而能够对超出主内存分配的数据集进行高性能操作。虽然键/值存储往往是内部状态存储最常见的实现,但可以使用任何形式的数据存储。关系或文档数据存储实现并非闻所未闻,但同样,它需要实例化并包含在每个单独的微服务实例中。
High-performance key/value stores, such as RocksDB, are typically used to implement internal state stores and are optimized to be highly efficient with local solid-state drives (SSDs), enabling performant operations on data sets that exceed main memory allocation. While key/value stores tend to be the most common implementation for internal state stores, any form of data store can be used. A relational or document data store implementation would not be unheard of, but again, it would need to be instantiated and contained within each individual microservice instance.
全局状态存储是内部状态存储的一种特殊形式。全局状态存储不是仅具体化分配给它的分区,而是具体化给定事件流的所有分区的数据,从而为每个微服务实例提供事件数据的完整副本。图 7-4说明了全局物化状态和非全局物化状态之间的区别。
A global state store is a special form of the internal state store. Instead of materializing only the partitions assigned to it, a global state store materializes the data of all partitions for a given event stream, providing a complete copy of the event data to each microservice instance. Figure 7-4 illustrates the difference between global and nonglobal materialized state.
当每个实例都需要完整的数据集并且往往包含小型、常用且很少更改的数据集时,全局状态存储非常有用。全局物化不能有效地用作事件驱动逻辑的驱动程序,因为每个微服务实例都拥有数据的完整副本,因此会产生重复的输出和不确定的结果。因此,最好为公共数据集查找和维度表保留全局具体化。
Global state stores are useful when a full data set is required by each instance, and tend to comprise small, commonly used, and seldom-changing data sets. Global materialization cannot be effectively used as the driver for event-driven logic, as each microservice instance possesses a full copy of the data and thus would produce duplicate outputs and nondeterministic results. For this reason, it is best to reserve global materialization for common data set lookup and dimension tables.
在本地磁盘上使用内部状态存储的一个主要好处是所有可扩展性要求都完全卸载到事件代理和计算资源集群。这使得应用程序开发团队能够严格专注于编写应用程序逻辑,同时依靠微服务能力团队提供所有事件驱动的微服务通用的扩展机制。这种方法确保了单个单元的可扩展性,其中每个应用程序都可以通过增加和减少实例计数来扩展。
A major benefit of using internal state stores on local disk is that all scalability requirements are fully offloaded to the event broker and compute resource clusters. This allows the application development team to focus strictly on writing application logic while relying on the microservices capability teams to provide the scaling mechanisms common to all event-driven microservices. This approach ensures a single unit of scalability, where each application can be scaled simply by increasing and decreasing the instance count.
当您考虑内部状态存储时,了解应用程序的性能要求非常重要。对于现代云计算,本地磁盘不一定意味着物理连接的磁盘,因为网络连接存储可以模仿本地磁盘并为您的应用程序提供相同的逻辑支持。高吞吐量的有状态流微服务每秒可以轻松消耗数十万个事件。您必须仔细考虑应用程序所需的性能特征,以确保其能够满足其延迟要求。
It’s important to understand your application’s performance requirements when you’re considering internal state storage. With modern cloud computing, local disk does not necessarily mean disk that is physically attached, as network-attached storage can mimic local disk and provide the same logical support for your applications. A high-throughput stateful streaming microservice can easily consume hundreds of thousands of events per second. You must carefully consider the performance characteristics required by the application to ensure it can meet its latency requirements.
在事件驱动的微服务中,维护主内存中的所有状态并不总是可能的,特别是在要保持较低成本的情况下。对于大多数现代微服务用例来说,物理连接的本地磁盘可以具有相当高的性能。本地磁盘实现倾向于采用高随机访问模式,通常由 SSD 支持。例如,使用 RocksDB 从 SSD 进行随机访问读取的延迟约为 65 微秒,这意味着单个线程的顺序访问上限约为 15.4k 请求/秒。内存性能明显更快,每秒可处理数百万个随机访问请求作为常态。本地磁盘和本地内存方法可实现极高的吞吐量,并显着减少数据访问瓶颈。
Maintaining all state within main memory is not always possible in an event-driven microservice, especially if costs are to be kept low. Physically attached local disk can be quite performant for the majority of modern microservice use cases. Local disk implementations tend to favor high random-access patterns, generally supported by SSDs. For instance, the latency for a random-access read from an SSD using RocksDB is approximately 65 microseconds, which means a single thread will have a sequential access ceiling of approximately 15.4k requests/second. In-memory performance is significantly faster, serving millions of random-access requests per second as the norm. The local-disk and local-memory approach allows for extremely high throughput and significantly reduces the data-access bottleneck.
微服务还可能使用网络连接磁盘而不是本地磁盘,这会显着增加读/写延迟。由于事件通常必须一次处理一个事件以维持时间和偏移顺序,因此单个处理线程将花费大量时间等待读/写响应,从而导致每个处理器的吞吐量显着降低。对于不需要高性能处理的任何有状态服务来说,这通常没问题,但如果事件量很大,则可能会出现问题。
Microservices may also use network-attached disk instead of local disk, which significantly increases the read/write latency. Since events typically must be processed one at a time to maintain temporal and offset order, the single processing thread will spend a lot of time awaiting read/write responses, resulting in significantly lower throughput per processor. This is generally fine for any stateful service that doesn’t need high-performance processing, but can be problematic if event volumes are high.
访问存储在网络连接磁盘上的“本地”数据比访问存储在系统内存或连接磁盘中的物理本地数据具有更高的延迟。虽然与本地 SSD 配对的 RocksDB 的估计吞吐量为 15.4k 请求/秒,但在相同的访问模式中引入仅 1 毫秒往返时间的网络延迟可将吞吐量上限降低至仅 939 请求/秒。虽然您可能能够做一些工作来并行化访问并减少这种差距,但请记住,事件必须按照消耗事件的偏移顺序进行处理,并且在许多情况下并行化是不可能的。
Accessing “local” data stored on network-attached disk has a much higher latency than accessing physically local data stored in the system’s memory or attached disk. While RocksDB paired with a local SSD has an estimated throughput of 15.4k request/second, introducing a network latency of only 1 mS round-trip time to an identical access pattern reduces the throughput cap to just 939 requests/second. While you might be able to do some work to parallelize access and reduce this gap, remember that events must be processed in the offset sequence in which they are consumed and that parallelization is not possible in many cases.
网络连接磁盘的一大优点是可以在卷中维护状态并根据需要迁移到新的处理硬件。当处理节点恢复时,可以重新连接网络磁盘,并且处理可以从中断处恢复,而不是从更改日志流重建。这大大减少了停机时间,因为状态不再像本地磁盘那样完全是短暂的,并且还提高了微服务跨计算资源迁移的灵活性,例如当您使用廉价的按需节点时。
One major benefit of network-attached disk is that the state can be maintained in the volume and migrated to new processing hardware as needed. When the processing node is brought back up, the network disk can be reattached and processing can resume where it left off, instead of being rebuilt from the changelog stream. This greatly reduces downtime since the state is no longer completely ephemeral as with a local disk, and also increases the flexibility of microservices to migrate across compute resources, such as when you are using inexpensive on-demand nodes.
内部状态存储仅限于使用在服务运行时定义并附加到节点的磁盘。更改附加卷的大小或数量通常需要停止服务、调整卷并重新启动服务。此外,许多计算资源管理解决方案仅允许增加卷大小,因为减少卷大小意味着需要删除数据。
Internal state stores are limited to using only disk that is defined and attached to the node at the service’s runtime. Changing either the size or quantity of attached volumes typically requires that the service be halted, the volumes adjusted, and the service restarted. In addition, many compute resource management solutions allow only for volume size to be increased, as decreasing a volume’s size means that data would need to be deleted.
本质上是周期性的数据模式,例如购物网站在下午 3 点和凌晨 3 点产生的流量,可能需要周期性存储量。也就是说,这些模式可能需要较大的最大磁盘来满足峰值流量,但其他情况下只需要少量磁盘。与使用仅按存储数据字节收费的外部服务相比,始终保留完整磁盘可能会浪费空间和金钱。
Data patterns that are cyclical in nature, such as the traffic generated to a shopping website at 3 p.m. versus 3 a.m., can require cyclical storage volume. That is, these patterns may require a large maximum disk for peak traffic but only a small amount otherwise. Reserving full disk for the entire time can waste both space and money when compared to using external services that charge you only per byte of data stored.
从状态恢复的角度来看,将处理扩展到多个实例和恢复失败的实例是相同的过程。新的或恢复的实例需要具体化其拓扑定义的任何状态,然后才能开始处理新事件。最快的方法是为应用程序中具体化的每个有状态存储重新加载变更日志主题。
Scaling processing up to multiple instances and recovering a failed instance are identical processes from a state recovery perspective. The new or recovered instance needs to materialize any state defined by its topology before it can begin processing new events. The quickest way to do so is to reload the changelog topic for each stateful store materialized in the application.
虽然最常见的是每个分区只有一个物化状态副本,但可以通过一些仔细的状态管理来创建其他副本或由客户端框架直接利用。Apache Kafka通过简单的配置设置将此功能内置到其 Streams 框架中。此设置提供高度可用的状态存储,并使微服务能够以零停机时间容忍实例故障。
While it is most common to have only a single replica of materialized state for each partition, additional replicas can be created through some careful state management or leveraged directly by the client framework. Apache Kafka has this functionality built into its Streams framework via a simple configuration setting. This setting provides highly available state stores and enables the microservice to tolerate instance failures with zero downtime.
图 7-5显示了内部状态存储复制因子为 2 的三实例部署。每个有状态分区被物化两次,一次作为领导者,一次作为副本。每个副本必须管理自己的偏移量,以确保它与领导者副本的偏移量保持一致。实例 0 和实例 1 正在处理流 B 事件并将它们加入到共同分区的物化状态上。实例 1 和实例 2 还分别维护流 A-P0 和 A-P1 的热副本,否则实例 2 不处理任何其他事件。
Figure 7-5 shows a three-instance deployment with an internal state store replication factor of 2. Each stateful partition is materialized twice, once as the leader and once as a replica. Each replica must manage its own offsets to ensure that it is keeping up with the offsets of the leader replica. Instance 0 and instance 1 are processing stream B events and joining them on the copartitioned materialized state. Instance 1 and instance 2 are also maintaining hot replicas of stream A-P0 and A-P1, respectively, with instance 2 otherwise not processing any other events.
当领导者终止时,消费者组必须重新平衡分区的分配。分区分配器确定热副本的位置(它之前分配了所有分区并知道所有分区到实例的映射)并相应地重新分配分区。在图 7-6中,实例 1 已终止,其余微服务实例被迫重新平衡其分区分配。具有热副本的实例将优先获得分区所有权并立即恢复处理。分区分配器已选择实例 2 来恢复流 B-P1 的处理。
When the leader is terminated, the consumer group must rebalance the assignment of partitions. The partition assignor determines the location of the hot replica (it previously assigned all partitions and knows all partition-to-instance mappings) and reassigns the partitions accordingly. In Figure 7-6, instance 1 has terminated, and the remaining microservice instances are forced to rebalance their partition assignments. Instances with hot replicas are given priority to claim partition ownership and resume processing immediately. The partition assignor has selected instance 2 to resume processing of stream B-P1.
一旦处理恢复,必须根据变更日志构建新的热副本以维持最小副本数量。构建新的热副本并将其添加到剩余实例中,如图7-7所示。
Once processing has resumed, new hot replicas must be built from the changelog to maintain the minimum replica count. The new hot replicas are built and added to the remaining instances, as shown in Figure 7-7.
当新创建的微服务实例加入消费者组时,只需从其变更日志中消费即可重新加载为其分配的任何有状态分区。在此期间,实例不应处理新事件,因为这样做可能会产生不确定的错误结果。
When a newly created microservice instance joins the consumer group, any stateful partitions that it is assigned can be reloaded simply by consuming from its changelog. During this time the instance should not be processing new events, as doing so could produce nondeterministic and erroneous results.
如果没有维护变更日志,微服务实例可以从输入流重建其状态存储。它必须从分配的事件流分区的最开始重新使用所有输入事件。每个事件都必须按照严格的递增顺序使用和处理,更新其状态,并生成任何后续输出事件。
If no changelog is maintained, the microservice instance can rebuild its state stores from the input streams. It must re-consume all of its input events from the very beginning of its assigned event stream partitions. Each event must be consumed and processed in strict incrementing order, its state updated, and any subsequent output events produced.
考虑完全再处理过程中产生的事件的影响。下游消费者可能需要幂等地处理这些或将它们作为重复项消除。
Consider the impact of events produced during a full reprocessing. Downstream consumers may need to process these idempotently or eliminate them as duplicates.
与从更改日志恢复相比,此过程重建状态可能需要更长的时间。因此,最好仅对简单拓扑采用此策略,其中重复输出不是问题、输入事件流保留时间较短且实体事件流稀疏。
This process can take much longer to rebuild state than restoring from a changelog. So, it’s best to employ this strategy only for simple topologies where duplicate output is not a concern, input event stream retention is short, and the entity event streams are sparse.
外部状态存储存在于微服务的容器或虚拟机之外,但通常位于同一本地网络内。您可以使用您喜欢的技术来实现外部数据存储,但您应该根据微服务问题空间的需求来选择它。外部存储服务的一些常见示例包括关系数据库;文档数据库;基于 Lucene 的地理空间搜索系统;以及分布式、高度可用的键/值存储。
External state stores exist outside of a microservice’s container or virtual machine, but are typically located within the same local network. You can implement an external data store using your preferred technology, but you should select it based on the needs of the microservice’s problem space. Some common examples of external store services include relational databases; document databases; a Lucene-based geospatial search system; and distributed, highly available key/value stores.
请记住,虽然特定微服务的外部状态存储可能使用通用数据存储平台,但数据集本身必须在逻辑上与所有其他微服务实现保持隔离。对于寻求使用通用物化数据集来满足多种业务需求的外部数据存储实现者来说,在微服务之间共享物化状态是一种常见的反模式。这可能会导致完全不相关的产品或功能之间的紧密耦合,应该避免。
Keep in mind that while a specific microservice’s external state store may use a common data storage platform, the data set itself must remain logically isolated from all other microservice implementations. Sharing materialized state between microservices is a common anti-pattern for implementers of external data stores who seek to use a common materialized data set to serve multiple business needs. This can lead to tight coupling between otherwise completely unrelated products or features and should be avoided.
不要与其他微服务共享直接状态访问。相反,所有微服务都必须实现自己的状态副本。这消除了直接耦合,并将微服务与无意的更改隔离开来,但代价是额外的处理和数据存储资源。
Do not share direct state access with other microservices. Instead, all microservices must materialize their own copy of state. This eliminates direct couplings and isolates microservices against unintentional changes, but at the expense of extra processing and data storage resources.
与内部状态存储不同,外部状态存储可以提供对每个微服务实例的所有物化数据的访问,尽管每个实例仍然负责物化其自己分配的分区。当您在外键上执行查找、关系查询以及大量元素之间的地理空间查询时,单个物化数据集无需分区局部性。
Unlike internal state stores, external state stores can provide access to all materialized data for each microservice instance, though each instance is still responsible for materializing its own assigned partitions. A single materialized data set eliminates the need for partition locality when you are performing lookups, relational queries on foreign keys, and geospatial queries between a large number of elements.
使用具有强大的先写后读保证的状态存储可以消除使用多个实例时不一致的结果。
Use state stores with strong read-after-write guarantees to eliminate inconsistent results when using multiple instances.
外部数据存储可以利用组织已经熟悉的技术,减少将微服务交付到生产所需的时间和精力。基本的消费者/生产者模式特别适合使用外部数据存储,如第 10 章所述。功能即服务解决方案也是优秀的外部数据存储候选者,如第 9 章所述。
External data stores can leverage technology that the organization is already familiar with, reducing the time and effort it takes to deliver a microservice to production. Basic consumer/producer patterns are especially good candidates for using external data stores, as covered in Chapter 10. Function-as-a-Service solutions are also excellent external data store candidates, as covered in Chapter 9.
外部状态存储的管理和扩展独立于微服务业务逻辑解决方案。外部数据存储的风险之一是微服务所有者现在需要确保其得到适当的维护和扩展。每个团队必须实施适当的资源分配、扩展策略和系统监控,以确保其数据服务适合微服务的负载并具有弹性。由组织的能力团队或第三方云平台供应商提供的托管数据服务可以帮助分配部分责任。
External state stores are managed and scaled independently of the microservice business logic solution. One of the risks of an external data store is that the microservice owner is now on the hook for ensuring that it is maintained and scaled appropriately. Each team must implement proper resource allocation, scaling policies, and system monitoring to ensure that their data service is suitable and resilient for the microservice’s load. Having managed data services provided by the organization’s capabilities team or by a third-party cloud platform vendor can help distribute some of this responsibility.
每个团队必须完全管理其微服务的外部状态存储。不要将外部状态存储管理的责任委托给其自己的团队,因为这会引入技术上的跨团队依赖性。撰写可接受的外部数据服务列表,并提供有关如何正确管理和扩展这些服务的指南。这将防止每个团队必须独立发现自己的管理解决方案。
Each team must fully manage the external state stores for its microservices. Do not delegate responsibility of external state store management to its own team, as this introduces a technical cross-team dependency. Compose a list of acceptable external data services with guides on how to properly manage and scale them. This will prevent each team from having to independently discover its own management solutions.
访问存储在外部状态存储中的数据比访问本地存储在内存或磁盘上的数据具有更高的延迟。在“使用内部状态的优点”中,您看到使用网络连接磁盘会带来轻微的网络延迟,并且会显着降低吞吐量和性能。
Accessing data stored in an external state store has a much higher latency than accessing data stored locally in memory or on disk. In “Advantages of Using Internal State”, you saw that using a network-attached disk introduces a slight network delay and can significantly reduce throughput and performance.
虽然缓存和并行化可以减少网络延迟的影响,但代价往往是增加复杂性以及额外内存和 CPU 的成本。并非所有微服务模式都支持缓存和并行化工作,许多微服务模式要求处理线程简单地阻塞并等待来自外部数据存储的答复。
While caching and parallelization may reduce the impact of the network latency, the tradeoff is often added complexity and an increased cost for additional memory and CPU. Not all microservice patterns support caching and parallelization efforts either, with many requiring the processing thread to simply block and wait for a reply from the external data store.
外部数据存储的财务成本往往比类似大小的内部数据存储更高。托管外部状态存储解决方案通常按事务数量、数据负载大小和数据保留期限收费。它们还可能需要过度配置来处理突发和不一致的负载。具有灵活性能特征的按需定价模型可能有助于降低成本,但您必须确保它们仍然满足您的性能需求。
Financial costs tend to be higher with external data stores than with similarly sized internal data stores. Hosted external state store solutions often charge by the number of transactions, the size of the data payload, and the retention period for the data. They may also require over-provisioning to handle bursty and inconsistent loads. On-demand pricing models with flexible performance characteristics may help reduce costs, but you must be sure they still meet your performance needs.
尽管也被列为一项优势,但完整的数据局部性可能会带来一些挑战。外部状态存储中的可用数据来自多个处理器和多个分区,每个处理器都以自己的速率进行处理。推断(和调试)任何特定处理实例对集体共享状态的贡献可能很困难。
Though also listed as a benefit, full data locality can present some challenges. The data available in the external state store originates from multiple processors and multiple partitions, each of which is processing at its own rate. It can be difficult to reason about (and debug) the contributions of any particular processing instance to the collective shared state.
还必须小心避免竞争条件和不确定性行为,因为每个微服务实例都在其自己独立的流时间上运行。单个微服务实例的流时间保证不会扩展到所有微服务实例。
Race conditions and nondeterministic behavior must also be carefully avoided, as each microservice instance operates on its own independent stream time. The stream time guarantees of a single microservice instance do not extend to all of them.
例如,一个实例可能会尝试加入尚未由单独实例填充的外键上的事件。稍后重新处理相同的数据可能会执行连接。由于每个实例的流处理与其他实例完全分开,因此通过此方法获得的任何结果都可能是不确定的且不可再现的。
For example, one instance may attempt to join an event on a foreign key that has not yet been populated by a separate instance. Reprocessing the same data at a later time may execute the join. Because the stream processing of each instance is completely separate from the others, any results obtained through this approach are likely to be nondeterministic and nonreproducible.
使用外部状态存储扩展和恢复微服务只需添加一个具有访问状态存储所需凭据的新实例。相反,底层状态存储的扩展和恢复完全取决于所选的技术,并且要复杂得多。
Scaling and recovery of microservices using an external state store simply require that you add a new instance with the necessary credentials to access the state store. In contrast, scaling and recovery of the underlying state store are dependent completely upon the selected technology and are much more complicated.
重申一下之前的观点,拥有一份可接受的外部数据服务列表以及如何正确管理、扩展、备份和恢复这些服务的指南对于为开发人员提供可持续的前进方向至关重要。不幸的是,状态存储技术的数量过多,实际上不可能在本书中进行讨论。相反,我将简单地将构建状态的策略概括为三种主要技术:从源流重建、使用更改日志和创建 快照。
To reiterate an earlier point, having a list of acceptable external data services with guides on how to properly manage, scale, back up, and restore them is essential for providing developers a sustainable way forward. Unfortunately, the number of state store technologies is prohibitively large and effectively impossible to discuss in this book. Instead, I’ll simply generalize the strategies of building state into three main techniques: rebuilding from the source streams, using changelogs, and creating snapshots.
通过从源流开始消耗事件来创建状态存储的新副本。对于所有输入流,消费者组输入偏移量都会回滚到时间的开头。此方法会导致所有选项中最长的停机时间,但易于重现,并且仅依赖事件代理的持久存储来维护源数据。请记住,此选项实际上是一个完整的应用程序重置,并且还将导致根据微服务的业务逻辑复制任何输出事件。
Created by consuming events from the beginning of time from the source streams creates a fresh copy of the state store. The consumer group input offsets are rewound to the beginning of time for all input streams. This method incurs the longest downtime of all options, but is easily reproducible and relies only on the persistent storage of the event broker to maintain the source data. Keep in mind that this option is really a full application reset and will also result in the reproduction of any output events according to the business logic of the microservice.
外部状态存储通常不依赖于使用代理存储的更改日志来记录和恢复状态,尽管没有规则阻止这样做。与内部状态存储非常相似,外部状态存储可以从变更日志中重新填充。就像从源流重建时一样,您必须创建状态存储的新副本。如果从变更日志重建,微服务使用者实例必须确保在恢复处理之前重建存储在变更日志中的整个状态。
External state stores typically do not rely on using broker-stored changelogs to record and restore state, though there is no rule preventing this. Much like an internal state store, external state stores can be repopulated from a changelog. Just like when rebuilding from source streams, you must create a fresh copy of the state store. If rebuilding from changelogs, the microservice consumer instances must ensure they rebuild the entire state as stored in the changelog before resuming processing.
由于网络延迟开销,从源事件流或变更日志重建外部状态存储可能非常耗时。确保在这种情况下您仍然可以满足微服务 SLA。
Rebuilding external state stores from source event streams or changelogs can be prohibitively time-consuming due to network latency overhead. Make sure you can still meet the microservice SLAs in such a scenario.
外部状态存储提供自己的备份和恢复过程更为典型,并且许多托管状态存储服务提供简单的“一键式”解决方案来实现此目的。应根据给定的状态存储实现遵循捕获和恢复状态的最佳实践。
It is far more typical for external state stores to provide their own backup and restoration process, and many hosted state store services provide simple “one-click” solutions to do so. Best practices for capturing and restoring state should be followed depending on the given state store implementation.
如果存储的状态是幂等的,则无需确保偏移量与物化状态精确一致。在这种情况下,将消费者偏移量设置为拍摄快照前几分钟的值应确保不会丢失任何数据。这也确保了事件的处理具有“至少一次”的 保证。
If the stored state is idempotent, there is no need to ensure that the offsets are precisely in line with the materialized state. In this case, setting the consumer offsets to values from a few minutes before the snapshot was taken should ensure that no data is missed. This also ensures that events are processed with an “at-least-once” guarantee.
如果存储的状态不是幂等的并且任何重复事件都是不可接受的,那么您应该将使用者的分区偏移量与数据存储中的数据一起存储。这确保了消费者偏移量和关联状态是一致的。当从快照恢复状态时,消费者可以将其消费者组偏移设置为从创建快照的确切时间开始在快照中找到的偏移量。“维护一致状态”对此进行了更详细的介绍。
If the stored state is not idempotent and any duplicate events are not acceptable, then you should store your consumer’s partition offsets alongside the data within the data store. This ensures that the consumer offsets and the associated state are consistent. When state is restored from the snapshot, the consumer can set its consumer group offsets to those found in the snapshot from the exact time that the snapshot was created. This is covered in more detail in “Maintaining consistent state”.
对现有状态存储数据结构的更改经常需要伴随新的业务需求。微服务可能需要向现有事件添加新信息,与另一个物化表执行一些额外的联接步骤,或者以其他方式存储新派生的业务数据。在这种情况下,需要通过重建或迁移来更新现有状态存储以反映数据。
Changes to existing state store data structures frequently need to accompany new business requirements. A microservice may need to add new information to existing events, perform some extra join steps with another materialized table, or otherwise store newly derived business data. In such an event, the existing state store will need to be updated to reflect the data, either through rebuilding or migration.
重建微服务的状态存储通常是更新应用程序内部状态的最常见方法。微服务首先停止,消费者输入流偏移重置到开头。任何中间状态,例如存储在变更日志中或位于外部状态存储中的状态,都必须删除。最后,启动新版本的微服务,并在从输入事件流读回事件时重建状态。这种方法确保状态完全按照新业务逻辑的指定构建。所有新的输出事件也会被创建并向下游传播到订阅消费者。这些不被视为重复事件,因为业务逻辑和输出格式可能已更改,并且这些更改必须向下游传播。
Rebuilding the microservice’s state stores is typically the most common method of updating the internal state of an application. The microservice is first stopped, with the consumer input stream offsets reset to the beginning. Any intermediate state, such as that stored in a changelog or located in the external state store, must be deleted. Lastly, the new version of the microservice is started up, and the state is rebuilt as events are read back in from the input event streams. This approach ensures that the state is built exactly as specified by the new business logic. All new output events are also created and propagated downstream to the subscribing consumers. These are not considered duplicate events, as the business logic and output format may have changed, and these changes must be propagated downstream.
重建状态要求所有必要的输入事件流事件仍然存在,特别是需要状态和聚合具体化的任何事件。如果您的应用程序严重依赖一组输入数据,则必须确保此类源数据在微服务实现的数据存储之外随时可用。
Rebuilding state requires that all necessary input event stream events still exist, particularly anything that requires materialization of state and aggregations. If your application is critically reliant upon a set of input data, you must ensure that such source data is readily available outside of your microservice implementation’s data store.
重建需要时间,在微服务的 SLA 中考虑到这一点非常重要。练习重建的主要好处之一是,它可以通过运行微服务失败且所有状态丢失时所需的恢复过程来帮助您测试灾难恢复准备情况。
Rebuilding takes time, and it’s important to account for that in the microservice’s SLA. One of the main benefits of practicing rebuilding is that it helps you test your disaster recovery preparedness by running through the recovery process required when a microservice fails and all state is lost.
最后,某些业务需求绝对要求您从一开始就重新处理数据,例如提取仅存在于输入事件中的字段。除了重播输入事件之外,您无法通过任何其他方式获取此数据,此时重建状态存储是唯一可行的选择。
Finally, some business requirements absolutely require you to reprocess data from the beginning of time, such as those that extract fields that are present only in the input events. You cannot obtain this data in any other way than replaying the input events, at which point rebuilding the state store is the only viable option.
与更改的影响相比,大型状态存储可能需要很长时间才能重建,或者可能导致极其昂贵的数据传输成本。例如,考虑业务需求更改,其中要向微服务的输出事件流添加一个附加但可选的字段。此更改可能需要您向微服务的状态存储添加另一个列或字段。但是,企业可能不需要重新处理旧数据,而只想将逻辑应用于未来的新输入事件。对于由关系数据库支持的状态存储,您只需要更新业务逻辑以及关联的表定义。您可以简单地插入一个具有可为空默认值的新列,经过一系列快速测试后,您可以重新部署该应用程序。
Large state stores can take a long time to rebuild or can result in prohibitively expensive data transfer costs when compared to the impact of the change. For instance, consider a business requirement change where an additional, yet optional, field is to be added to a microservice’s output event stream. This change may require you to add another column or field to the microservice’s state store. However, it could be that the business has no need to reprocess older data and wants only to apply the logic to new input events going forward. For a state store backed by a relational database, you’d just need to update the business logic alongside the associated table definition. You can perform a simple insertion of a new column with a nullable default, and after a quick series of tests you can redeploy the application.
当业务需求和要更改的数据更加复杂时,迁移的风险就会变得更大。复杂的迁移很容易出错,并且与完全重建数据存储的结果相比,可能会产生不正确的结果。数据库迁移逻辑不是业务逻辑的一部分,因此可能会引入不一致,而这些不一致在应用程序的完全重建期间不会出现。如果在测试期间未发现这些类型的迁移错误,则很难检测到,并且可能导致数据不一致。当采用基于迁移的方法时,请务必执行严格的测试并使用代表性测试数据集将该方法与基于重建的方法进行比较。
Migrations become much riskier when the business need and the data being changed are more complex. Complex migrations are error-prone and can yield results that are incorrect compared to the results of a complete rebuild of the data store. The database migration logic is not a part of the business logic, so inconsistencies can be introduced that would otherwise not arise during a full rebuild of the application. These types of migration errors can be hard to detect if not caught during testing and can lead to inconsistent data. When following a migration-based approach, be sure to perform strict testing and use representative test data sets to compare that approach with a rebuild-based one.
有效的一次处理可确保一致地应用对单一事实源所做的任何更新,无论生产者、消费者或事件代理是否出现任何故障。有效一次处理有时也被描述为恰好一次处理,尽管这不太准确。微服务可能会多次处理相同的数据,例如由于消费者故障和随后的恢复,但无法提交其偏移量并增加其流时间。每次处理事件时都会执行处理逻辑,包括代码可能产生的任何副作用,例如将数据发布到外部端点或与第三方服务通信。话虽这么说,对于大多数事件代理和大多数用例来说,这些术语恰好一次和有效一次可以互换使用。
Effectively once processing ensures that any updates made to the single source of truth are consistently applied, regardless of any failure to the producer, the consumer, or the event broker. Effectively once processing is also sometimes described as exactly once processing, though this is not quite accurate. A microservice may process the same data multiple times, say due to a consumer failure and subsequent recovery, but fail to commit its offsets and increment its stream time. The processing logic will be executed each time the event is processed, including any side effects that the code may create, such as publishing data to external endpoints or communicating with a third-party service. That being said, for most event brokers and most use cases, the terms exactly once and effectively once are used interchangeably.
幂等写入是 Apache Kafka 和 Apache Pulsar 等事件代理实现中普遍支持的功能之一。它们允许将事件写入事件流一次且仅一次。如果生产者或事件代理在写入期间失败,幂等写入功能可确保重试时不会创建该事件的副本。
Idempotent writes are one commonly supported feature among event broker implementations such as Apache Kafka and Apache Pulsar. They allow for an event to be written once, and only once, to an event stream. In the case that the producer or event broker fails during the write, the idempotent write feature ensures that a duplicate of that event is not created upon retry.
您的事件代理也可能支持交易。目前,仅 Apache Kafka 提供完整的事务支持,尽管 Apache Pulsar 正在自行实现方面取得进展。就像关系数据库可以支持单个事务中的多表更新一样,事件代理实现也可以支持将多个事件原子写入多个单独的事件流。这允许生产者在单个原子事务中将其事件发布到多个事件流。缺乏事务支持的竞争事件代理实现要求客户端在处理后确保其自身有效。下一节将介绍这两个选项,并评估如何将它们用于您自己的微服务。
Transactions may also be supported by your event broker. Currently, full transactional support is offered only by Apache Kafka, though Apache Pulsar is making progress towards its own implementation. Much like a relational database can support multitable updates in a single transaction, an event broker implementation may also support the atomic writing of multiple events to multiple separate event streams. This allows a producer to publish its events to multiple event streams in a single, atomic transaction. Competing event broker implementations that lack transactional support require that the client ensure its own effectively once processing. The next section covers both of these options and evaluates how you can leverage them for your own microservices.
事务功能极其强大,使 Apache Kafka 比竞争对手具有显着优势。特别是,它们可以满足新的业务需求,否则这些需求将需要复杂的重构来确保原子生产。
Transactions are extremely powerful and give Apache Kafka a significant advantage over its competitors. In particular, they can accommodate new business requirements that would otherwise require a complex refactoring to ensure atomic production.
库存会计服务负责在任何给定商品的库存不足时发出通知事件。微服务必须根据随着时间推移进行的一系列增减,将每种产品的当前可用库存拼凑起来。向客户出售商品、因损坏而丢失商品以及因被盗而丢失商品都是减少库存的事件,而接收货物和接受客户退货会增加库存。为了简单起见,在本示例中这些事件显示在同一事件流中,如图7-8所示。
The stock accounting service is responsible for issuing a notification event when stock of any given item is low. The microservice must piece together the current stock available for each product based on a chain of additions and subtractions made over time. Selling items to customers, losing items to damage, and losing items to theft are all events that reduce stock, whereas receiving shipments and accepting customer returns increase it. These events are shown in the same event stream for simplicity in this example, as illustrated in Figure 7-8.
这种股票会计服务非常简单。它根据事件流变化计算当前运行的库存总量,并将其存储在数据存储中。业务逻辑根据阈值进行过滤,并决定是否向库存管理发出有关低库存或超卖库存的通知。必须确保每个输入事件有效地应用于聚合状态一次,因为多次应用它是不正确的,就像根本不应用它一样。一旦处理发挥作用,这就是有效的地方。
This stock accounting service is quite simple. It calculates the current running total of stock based off of the event stream changes and stores it in its data store. The business logic filters on a threshold value and decides whether to issue a notification to stock management about low or oversold stock. It must be ensured that each input event is applied effectively once to the aggregated state, as applying it more than once is incorrect, as is not applying it at all. This is where effectively once processing comes into play.
任何支持事务的事件代理都可以有效地促进处理。通过这种方法,任何输出事件、对由变更日志支持的内部状态进行的更新以及消费者偏移量的增量都被包装在单个原子事务中。仅当所有这三个更新都存储在代理中其自己的特定事件流中时,这才有可能。偏移量更新、变更日志更新和输出事件在单个事务中以原子方式提交,如图7-9所示。
Effectively once processing can be facilitated by any event broker that supports transactions. With this approach, any output events, updates made to internal state backed by a changelog, and the incrementing of the consumer offsets are wrapped together within a single atomic transaction. This is possible only if all three of these updates are stored within their own specific event stream in the broker. The offset update, the changelog update, and the output event are committed atomically within a single transaction as shown in Figure 7-9.
生产者客户端和事件代理之间的原子事务会将所有事件发布到其相应的事件流。在生产者永久失败的情况下,如图7-10所示,代理将确保事务中的任何事件都不会提交。事件流使用者通常不会处理未提交事务中的事件。消费者必须遵守偏移顺序,因此它将阻塞,等待事务完成,然后继续处理事件。在出现暂时性错误的情况下,生产者可以简单地重试提交其事务,因为它是幂等操作。
The atomic transaction between the producer client and the event broker will publish all events to their corresponding event streams. In the case of permanent failure by the producer, as in Figure 7-10, the broker will ensure that none of the events in the transaction is committed. Event stream consumers typically abstain from processing events that are in uncommitted transactions. The consumer must respect offset order, and so it will block, wait for the transaction to complete, and then proceed to process the event. In the case of transient errors, the producer can simply retry committing its transaction, as it is an idempotent operation.
如果生产者在事务期间遇到致命异常,则可以通过从更改日志中恢复来重建其替换实例,如图7-11所示。输入事件流的消费者组偏移量也会根据偏移事件流中存储的最后一个已知的良好位置进行重置。
In the case that the producer suffers a fatal exception during a transaction, its replacement instance can simply be rebuilt by restoring from the changelogs as shown in Figure 7-11. The consumer group offsets of the input event streams are also reset according to the last known good position stored in the offset event stream.
一旦生产者恢复,新的事务就可以开始,并且所有先前未完成的事务都会失败并由事件代理清理。事务机制会根据代理实现的不同而有所不同,因此请确保熟悉您正在使用的机制。
New transactions can begin once the producer is recovered, and all previous incomplete transactions are failed and cleaned up by the event broker. The transactional mechanisms will vary to some extent depending on the broker implementation, so make sure to familiarize yourself with the one you are using.
对于不支持客户端代理事务的实现,也可以有效地一次处理事件,尽管它需要更多的工作并仔细考虑重复事件。首先,如果上游服务不能有效提供一次事件生产保证,那么就有可能产生重复记录。需要识别并过滤掉上游流程创建的任何重复事件。其次,状态和偏移管理需要在本地事务中更新以确保事件处理仅应用于系统状态一次。通过遵循此策略,客户端可以确信其处理器生成的内部状态与输入事件流的逻辑叙述一致。让我们更详细地看看这些步骤。
Effectively once processing of events is also possible for implementations that do not support client-broker transactions, though it requires more work and a careful consideration of duplicate events. First, if upstream services are not able to provide effectively once event production guarantees, then it is possible that they may produce duplicate records. Any duplicate events created by upstream processes need to be identified and filtered out. Second, state and offset management need to be updated in a local transaction to ensure that the event processing is applied only once to the system state. By following this strategy, clients can be assured that the internal state generated by their processor is consistent with the logical narrative of the input event streams. Let’s take a look at these steps in more detail.
最好使用支持幂等写入的事件代理和客户端,而不是事后尝试解决重复数据删除问题。前一种方法可以很好地扩展到所有消费者应用程序,而后者则昂贵且难以扩展。
It is better to use an event broker and client that support idempotent writes than it is to try to solve deduplication after the fact. The former method scales well to all consumer applications, whereas the latter is expensive and difficult to scale.
当生产者成功将事件写入事件流,但无法接收写入确认并重试,或者在更新其自己的消费者偏移量之前崩溃时,就会生成重复事件。这些场景略有 不同:
Duplicate events are generated when a producer successfully writes the events to an event stream, but either fails to receive a write acknowledgment and retries, or crashes before updating its own consumer offsets. These scenarios are slightly different:
在这种情况下,生产者仍然在其内存中具有要生成的事件的副本。这些事件如果再次发布,可能具有相同的时间戳(如果它们使用事件创建时间)和相同的事件数据,但将被分配新的偏移量。
In this scenario, the producer still has the copies of the events to produce in its memory. These events, if published again, may have the same timestamps (if they use event creation time) and same event data, but will be assigned new offsets.
在这种情况下,生产者将成功写入其事件,但尚未更新其消费者偏移量。这意味着当生产者恢复时,它将重复之前所做的工作,创建逻辑上相同的事件副本,但具有新的时间戳。如果处理是确定性的,那么事件将具有相同的数据。还将分配新的偏移量。
In this case, the producer will have successfully written its events, but will not have updated its consumer offsets yet. This means that when the producer comes back up, it will repeat the work that it had previously done, creating logically identical copies of the events but with new timestamps. If processing is deterministic, then the events will have the same data. New offsets will also be assigned.
幂等生产受到众多事件代理的支持,并且可以减轻由于崩溃和重试而导致的故障,例如在前面的两个场景中。它无法减少由于错误的业务逻辑而引入的重复。
Idempotent production is supported by numerous event brokers and can mitigate failures due to crashes and retries, such as in the two preceding scenarios. It cannot mitigate duplicates introduced through faulty business logic.
如果事件的幂等生成不可用,并且事件流中存在重复项(具有唯一的偏移量和唯一的时间戳),则您需要减轻其影响。首先,确定重复项是否确实会导致任何问题。在许多情况下,重复项的影响即使不可忽略,也很小,可以忽略不计。对于重复事件确实会导致问题的情况,您需要弄清楚如何识别它们。实现此目的的一种方法是让生产者 为每个事件生成唯一的 ID,以便任何重复项都将生成相同的唯一哈希值。
If idempotent production of events is not available and there are duplicates (with unique offsets and unique timestamps) in the event stream, then it is up to you to mitigate their impact. First, determine if the duplicates actually cause any problems. In many cases duplicates have a minor, if not negligible, effect and can simply be ignored. For those scenarios where duplicate events do cause problems, you will need to figure out how to identify them. One way to do this is to have the producer generate a unique ID for each event, such that any duplicates will generate the same unique hash.
该哈希函数通常基于内部事件数据的属性,包括事件的键、值和创建时间。这种方法对于具有大数据域的事件往往效果很好,但对于逻辑上彼此等效的事件则效果不佳。以下是您可以生成唯一 ID 的几种情况:
This hash function is often based on the properties of the internal event data, including the key, values, and creation time of the event. This approach tends to work well for events that have a large data domain, but poorly for events that are logically equivalent to one another. Here are a few scenarios where you could generate a unique ID:
银行账户转账,详细说明来源、目的地、金额、日期和时间
A bank account transfer, detailing the source, destination, amount, date, and time
电子商务订单,详细说明每个产品、购买者、日期、时间、总金额和付款提供商
An ecommerce order, detailing each product, the purchaser, date, time, total amount, and payment provider
出于发货目的而借记的库存,其中每个事件都有一个关联的orderId(使用现有的唯一数据 ID)
Stock debited for shipment purposes, where each event has an associated orderId (uses an existing unique data ID)
这些示例的一个共同点是每个 ID 都由基数非常高(即唯一性)的元素组成。这显着降低了 ID 之间出现重复的可能性。重复数据删除ID(dedupe ID)可以随事件生成,也可以由消费者在消费时生成,前者更适合分发给所有消费者。
One factor these examples have in common is that each ID is composed of elements with a very high cardinality (that is, uniqueness). This significantly reduces the chances of duplicates between the IDs. The deduplication ID (dedupe ID) can either be generated with the event or be generated by the consumer upon consumption, with the former being preferable for distribution to all consumers.
防止没有密钥产生的重复事件极具挑战性,因为无法保证分区局部性。使用键生成事件,尊重分区局部性,并尽可能使用幂等写入。
Guarding against duplicate events produced without a key is extremely challenging, as there is no guarantee of partition locality. Produce events with a key, respect partition locality, and use idempotent writes whenever possible.
任何有效的一次消费者必须识别并丢弃重复项,执行幂等操作,或者从具有幂等生产者的事件流中进行消费。幂等操作并不适用于所有业务案例,如果没有幂等生产,您必须找到一种方法来保护业务逻辑免受重复事件的影响。这可能是一项昂贵的工作,因为它要求每个消费者维护先前处理的重复数据删除 ID 的状态存储。根据事件量以及应用程序必须防范的偏移量或时间范围,存储可能会变得非常大。
Any effectively once consumer must either identify and discard duplicates, perform idempotent operations, or consume from event streams that have idempotent producers. Idempotent operations are not possible for all business cases, and without idempotent production you must find a way to guard your business logic against duplicate events. This can be an expensive endeavor, as it requires that each consumer maintain a state store of previously processed dedupe IDs. The store can grow very large depending on the volume of events and the offset or time range that the application must guard against.
完美的重复数据删除要求每个使用者无限期地维护对已处理的每个重复数据删除 ID 的查找,但如果尝试防止范围过大,时间和空间要求可能会变得非常昂贵。实际上,重复数据删除通常仅针对特定的滚动时间窗口或偏移窗口执行,作为尽力而为的尝试。
Perfect deduplication requires that each consumer indefinitely maintain a lookup of each dedupe ID already processed, but time and space requirements can become prohibitively expensive if an attempt is made to guard against too large a range. In practice, deduplication is generally only performed for a specific rolling time-window or offset-window as a best-effort attempt.
通过使用生存时间 (TTL)、最大缓存大小和定期删除来保持重复数据删除存储较小。所需的具体设置将根据应用程序对重复的敏感性以及重复发生的影响而有所不同。
Keep deduplication stores small by using time-to-live (TTL), a maximum cache size, and periodic deletions. The specific settings needed will vary depending on the sensitivity of your application to duplicates and the impact of duplicates occurring.
应仅在单个事件流分区内尝试重复数据删除,因为分区之间的重复数据删除将非常昂贵。与非键控事件相比,键控事件具有额外的优势,因为它们将始终分布到同一分区。
Deduplication should be attempted only within a single event stream partition, as deduplication between partitions will be prohibitively expensive. Keyed events have an added benefit over unkeyed events, since they will consistently be distributed to the same partition.
图 7-12显示了正在运行的重复数据删除存储。在此图中,您可以看到事件在传递到实际业务逻辑之前所经历的工作流程。在本例中,TTL任意设置为8000秒,但实际中需要根据业务需求来确定。
Figure 7-12 shows a deduplication store in action. In this figure you can see the workflow that an event goes through before being passed off to the actual business logic. In this example the TTL is arbitrarily set to 8,000 seconds, but in practice would need to be established based on business requirements.
重复数据删除存储中使用最大高速缓存大小来限制所维护的事件数量,特别是在重新处理期间。
A maximum cache size is used in the deduplication store to limit the number of events maintained, particularly during reprocessing.
请注意,您有责任维护重复数据删除表的持久备份,就像任何其他物化表一样。如果发生故障,必须在恢复处理新事件之前重建表。
Note that you are responsible for maintaining durable backups of the deduplication table, just as for any other materialized table. In the case of a failure, the table must be rebuilt prior to resuming the processing of new events.
微服务可以利用其状态存储的事务功能而不是事件代理来有效执行一次处理。这需要将消费者组的偏移管理从事件代理转移到数据服务中,从而允许单个有状态事务以原子方式更新状态和输入偏移。对状态所做的任何更改与对消费者偏移量所做的更改完全一致,从而保持服务内的一致性。
A microservice can leverage the transactional capabilities of its state store instead of the event broker to perform effectively once processing. This requires moving the consumer group’s offset management from the event broker and into the data service, allowing a single stateful transaction to atomically update both the state and input offsets. Any changes made to the state coincide completely with those made to the consumer offsets, which maintains consistency within the service.
如果出现服务故障,例如提交数据服务超时,微服务可以简单地放弃事务并恢复到最后一个已知的良好状态。所有消耗都会停止,直到数据服务做出响应,此时消耗将从最后一个已知的良好偏移量恢复。通过使偏移量的官方记录与数据服务中的数据保持同步,您可以获得服务可以恢复的一致状态视图。该过程如图7-13、7-14和7-15所示。
In the case of a service failure, such as a timeout when committing to the data service, the microservice can simply abandon to transaction and revert to the last known good state. All consumption is halted until the data service is responsive, at which point consumption is restored from the last known good offset. By keeping the official record of offsets synchronized with the data in the data service, you have a consistent view of state that the service can recover from. This process is illustrated in Figures 7-13, 7-14, and 7-15.
请注意,此方法使您的处理器可以有效地进行一次处理,但不能有效地一次进行事件生成。此服务生成的任何事件都受到至少一次生成的限制,因为事件的非交易性客户端代理生成可能会创建重复项。
Note that this approach gives your processor effectively once processing, but not effectively once event production. Any events produced by this service are subject to the limitations of at-least-once production, as the nontransactional client-broker production of events is subject to the creation of duplicates.
如果您必须在更新状态存储的同时以事务方式生成事件,请参阅第 4 章。使用更改数据表可以为状态存储提供最终一致、有效的一次更新,并为输出事件流提供至少一次生成。
If you must have events transactionally produced alongside updates to the state store, refer to Chapter 4. Using a change-data table can provide eventually consistent, effectively once updates to the state store with at-least-once production to the output event stream.
本章介绍了内部和外部状态存储、它们如何工作、它们的优点、缺点以及何时使用它们。数据局部性在系统的延迟和吞吐量中发挥着重要作用,使系统能够在重负载时进行扩展。内部状态存储可以支持高性能处理,而外部状态存储可以提供一系列灵活的选项来支持微服务的业务需求。
This chapter looked at internal and external state stores, how they work, their advantages, their disadvantages, and when to use them. Data locality plays a large role in the latency and throughput of systems, allowing them to scale up in times of heavy load. Internal state stores can support high-performance processing, while external state stores can provide a range of flexible options for supporting the business needs of your microservices.
更改日志在微服务状态存储的备份和恢复中发挥着重要作用,尽管此作用也可以由事务支持数据库和定期计划的快照来执行。支持事务的事件代理可以实现极其强大的有效一次处理,从而减轻消费者的重复预防责任,而重复数据删除工作可以在没有这种支持的系统中有效地实现一次处理。
Changelogs play an important role in the backup and restoration of microservice state stores, though this role may also be performed by transaction-supporting databases and regularly scheduled snapshots. Event brokers that support transactions can enable extremely powerful effectively once processing, offloading the responsibility of duplication prevention from the consumer, while deduplication efforts can enable effectively once processing in systems without such support.
根据其定义,微服务仅在组织的整体业务工作流程的一小部分上运行。工作流是组成业务流程的一组特定操作,包括任何逻辑分支和补偿操作。工作流通常需要多个微服务,每个微服务都有自己的有界上下文,执行其任务并向下游消费者发出新事件。到目前为止,我们所了解的大部分内容都是单个微服务如何在后台运行。现在,我们将了解多个微服务如何协同工作来完成更大的业务工作流程,以及事件驱动的微服务方法中出现的一些陷阱和问题。
Microservices, by their very definition, operate on only a small portion of the overall business workflow of an organization. A workflow is a particular set of actions that compose a business process, including any logical branching and compensatory actions. Workflows commonly require multiple microservices, each with its own bounded context, performing its tasks and emitting new events to downstream consumers. Most of what we’ve looked at so far has been how single microservices operate under the hood. Now we’re going to take a look at how multiple microservices can work together to fulfill larger business workflows, and some of the pitfalls and issues that arise from an event-driven microservice approach.
以下是实施 EDM 工作流程的一些主要注意事项。
Here are some of the main considerations for implementing EDM workflows.
工作流程中的服务如何相关?
如何修改现有工作流程而不需要:
破坏工作已经在进行中吗?
需要更改多个微服务?
破坏监控和可见性?
How are the services related within the workflow?
How can I modify existing workflows without:
Breaking work already in progress?
Requiring changes to multiple microservices?
Breaking monitoring and visibility?
如何判断事件的工作流程何时完成?
如何判断事件是否无法处理或卡在 工作流程中的某个位置?
如何监控工作流程的整体运行状况?
How can I tell when the workflow is completed for an event?
How can I tell if an event has failed to process or is stuck somewhere in the workflow?
How can I monitor the overall health of a workflow?
许多工作流程要求许多操作同时发生或根本不发生。如何实现分布式事务?
如何回滚分布式事务?
Many workflows require that a number of actions happen together or not at all. How do I implement distributed transactions?
How do I roll back distributed transactions?
本章介绍了两种主要的工作流程模式:编排和编排,并根据这些考虑因素对它们进行评估。
This chapter covers the two main workflow patterns, choreography and orchestration, and evaluates them against these considerations.
术语编排架构(也称为反应式架构)通常指高度解耦的微服务架构。微服务在输入事件到达时对其做出反应,没有任何阻塞或等待,以完全独立于任何上游生产者或后续下游消费者的方式。这很像舞蹈表演,每个舞者必须知道自己的角色并独立表演,在舞蹈过程中不受控制或告诉要做什么。
The term choreographed architectures (also known as reactive architectures) commonly refers to highly decoupled microservice architectures. Microservices react to their input events as they arrive, without any blocking or waiting, in a manner fully independent from any upstream producers or subsequent downstream consumers. This is much like a dance performance, where each dancer must know his or her own role and perform it independently, without being controlled or told what to do during the dance.
编排在事件驱动的微服务架构中很常见。事件驱动架构严格专注于提供相关业务信息的可重用事件流,消费者可以在其中来来去去,而不会中断上游工作流程。所有通信都严格通过输入和输出事件流完成。精心设计的系统中的生产者不知道其数据的消费者是谁,也不知道他们打算执行什么业务逻辑或操作。上游业务逻辑与下游消费者完全隔离。
Choreography is common in event-driven microservice architectures. Event-driven architectures focus strictly on providing reusable event streams of relevant business information, where consumers can come and go without any disruption to the upstream workflow. All communications are done strictly through the input and output event streams. A producer in a choreographed system does not know who the consumers of its data are, nor what business logic or operations they intend to perform. The upstream business logic is fully isolated from the downstream consumers.
在大多数团队间通信中,编排是可取的,因为它允许松散耦合的系统并减少协调要求。许多业务工作流程彼此独立,不需要严格协调,这使得精心设计的方法非常适合沟通。新的微服务可以轻松添加到精心设计的架构中,而现有的微服务也可以轻松删除。
Choreography is desirable in the majority of interteam communications, as it allows for loosely coupled systems and reduces coordination requirements. Many business workflows are independent of one another and do not require strict coordination, which makes a choreographed approach ideal for communication. New microservices can be easily added to a choreographed architecture, while existing ones can be removed just as easily.
微服务之间的关系定义了精心设计的架构的工作流程。一系列一起运行的微服务可以负责提供工作流的业务功能。这种精心设计的工作流程是一种紧急行为,其中不仅决定工作流程的是单个微服务,还决定了它们之间的关系。
The relationships between the microservices define the workflow of a choreographed architecture. A series of microservices operating together can be responsible for providing the business functionality of the workflow. This choreographed workflow is a form of emergent behavior, where it is not just the individual microservices that dictate the workflow, but the relationships between them as well.
直接调用微服务架构专注于提供可重用的服务,用作业务工作流程的构建块。另一方面,事件驱动的微服务架构专注于提供可重用的事件,而不预知下游消费。后一种架构允许使用高度解耦、精心设计的架构。
Direct-call microservice architectures focus on providing reusable services, to be used as building blocks for business workflows. Event-driven microservice architectures, on the other hand, focus on providing reusable events, with no foreknowledge of downstream consumption. The latter architecture enables the usage of highly decoupled, choreographed architectures.
需要注意的是,编排属于事件驱动架构的领域,因为生产者服务和消费者服务的解耦使他们能够独立履行自己的职责。将此与直接调用微服务架构进行对比,直接调用微服务架构的重点是提供可重用的服务以组成更广泛的业务功能和工作流程,并且一个微服务直接调用另一个微服务的 API。根据定义,这意味着调用微服务必须知道两件事:
It is important to note that choreography belongs to the domain of event-driven architectures, because the decoupling of the producer and consumer services allows them to carry out their responsibilities independently. Contrast this with the direct-call microservice architecture, where the focus is on providing reusable services to compose more extensive business functionality and workflows, and where one microservice directly calls the API of another. By definition, this means the calling microservice must know two things:
需要调用哪个服务
Which service needs to be called
为什么需要调用服务(期望的业务价值)
Why the service needs to be called (the expected business value)
正如您所看到的,直接调用微服务是紧密耦合的,并且完全依赖于现有微服务的有界上下文。
As you can see, a direct-call microservice is tightly coupled and fully dependent on the existing microservice’s bounded contexts.
图 8-1显示了精心设计的工作流的输出,其中服务 A 直接馈送到服务 B,服务 B 又馈送到服务 C。在这种特殊情况下,您可以推断服务具有 A → B → C 的依赖工作流服务C的输出表示整个工作流程的结果。
Figure 8-1 shows the output of a choreographed workflow in which service A feeds directly into service B, which in turn feeds into service C. In this particular case, you can infer that the services have a dependent workflow of A → B → C. The output of service C indicates the result of the workflow as a whole.
现在,假设需要重新安排工作流程,使得服务 C 中的业务操作必须先于服务 B 中的业务操作执行,如图8-2所示。
Now, say that the workflow needs to be rearranged such that the business actions in service C must be performed before those in service B, as shown in Figure 8-2.
服务 C 和 B 都必须经过编辑才能分别从流 1 和 2 中使用。流中数据的格式可能不再适合新工作流程的需要,需要进行破坏性架构更改,这可能会显着影响事件流 2 的其他使用者(未显示)。可能需要为事件创建全新的事件架构流 2,事件流中的旧数据移植为新格式、删除或保留在原处。最后,在交换拓扑之前,必须确保服务 A、B、C 的所有输出事件都已完成处理,以免某些事件处理不完全而处于不一致的状态。
Both services C and B must be edited to consume from streams 1 and 2, respectively. The format of the data within the streams may no longer suit the needs of the new workflow, requiring breaking schema changes that may significantly affect other consumers (not shown) of event stream 2. A whole new event schema may need to be created for event stream 2, with the old data in the event stream ported over to the new format, deleted, or left in place. Finally, you must ensure that processing has completed for all of the output events of services A, B, and C before swapping the topology around, lest some events be incompletely processed and left in an inconsistent state.
虽然编排允许在工作流程末尾简单添加新步骤,但在中间插入步骤或更改工作流程的顺序可能会出现问题。在工作流上下文之外,服务之间的关系也可能难以理解,随着工作流中服务数量的增加,这一挑战会加剧。精心设计的工作流程可能很脆弱,特别是当业务功能跨多个微服务实例时。通过仔细尊重服务的有界上下文并确保完整的业务功能仍然位于单个服务的本地,可以缓解这种情况。然而,即使正确实现,小的业务逻辑更改也可能需要您修改或重写大量服务,尤其是那些更改工作流本身顺序的服务。
While choreography allows for simple addition of new steps at the end of the workflow, it may be problematic to insert steps into the middle or to change the order of the workflow. The relationships between the services may also be difficult to understand outside the context of the workflow, a challenge that is exacerbated as the number of services in the workflow increases. Choreographed workflows can be brittle, particularly when business functions cross multiple microservice instances. This can be mitigated by carefully respecting the bounded contexts of services and ensuring that full business functionality remains local to a single service. However, even when correctly implemented, small business logic changes may require you to modify or rewrite numerous services, especially those that change the order of the workflow itself.
在监控精心设计的工作流程时,您需要考虑其规模和范围。孤立地讲,分布式编排的工作流程可能会导致难以辨别特定事件的处理进度。对于事件驱动的系统,监视业务关键型工作流程可能需要侦听每个输出事件流并将其具体化到状态存储中,以说明事件可能无法处理或在处理过程中陷入困境的位置。
When monitoring a choreographed workflow, you need to take into account its scale and scope. In isolation, distributed choreographed workflows can make it difficult to discern the processing progress of a specific event. For event-driven systems, monitoring business-critical workflows may necessitate listening to each output event stream and materializing it into a state store, to account for where an event may have failed to process or have gotten stuck in processing.
例如,修改图 8-2中的工作流顺序还需要更改工作流可见性系统。这假设观察者关心该工作流中的每个事件流并且想要了解有关每个事件的所有信息。但是,如果您可能不关心工作流中的每个事件流,那么工作流的可见性又如何呢?遍布整个组织的工作流程又如何呢?
For example, modifying the workflow order in Figure 8-2 requires also changing the workflow visibility system. This assumes that observers care about each event stream in that workflow and want to know everything about each event. But what about visibility into workflows where you may not care about each event stream in the workflow? What about workflows spread out across an entire organization?
现在考虑一个更大规模的示例,例如大型跨国在线零售商的订单履行流程。客户查看商品、选择一些商品进行购买、付款并等待发货通知。可能有数十甚至数百个服务参与支持此客户工作流程。此工作流程的可见性将根据观察者的需求而变化。
Consider now a much larger-scale example, such as an order fulfillment process at a large multinational online retailer. A customer views items, selects some for purchase, makes a payment, and awaits a shipping notification. There may be many dozens or even hundreds of services involved in supporting this customer workflow. Visibility into this workflow will vary depending on the needs of the observer.
客户可能只真正关心订单从付款到履行再到发货通知的进展情况。您可以通过从三个单独的事件流中分出事件来合理地监视这一点。由于其“公共”性质和所消耗的事件流数量较少,它将具有足够的弹性来应对变化。同时,要了解完整的端到端工作流程可能需要消耗数十个事件流。由于事件量和事件流的独立性,这可能更具挑战性,特别是在工作流程经常发生变化的情况下。
The customer may only really care where the order is in the progression from payment to fulfillment to shipping notification. You could reasonably monitor this by tapping off events from three separate event streams. It would be sufficiently resilient to change due to both the “public” nature and the low number of event streams it is consuming from. Meanwhile, a view into the full end-to-end workflow could require consuming from dozens of event streams. This may be more challenging to accomplish due to both the volume of events and the independence of event streams, particularly if the workflow is subject to regular change.
确保您知道您想要在精心设计的工作流程中显示什么内容。不同的观察者有不同的要求,并且并非工作流程的所有步骤都需要显式暴露。
Be sure you know what it is you’re trying to make visible in the choreographed workflow. Different observers have different requirements, and not all steps of a workflow may require explicit exposure.
在编排模式中,中央微服务(编排器)向从属工作微服务发出命令并等待其响应。编排可以被认为很像一个管弦乐队,在表演过程中由一名指挥指挥音乐家。协调器微服务包含给定业务流程的整个工作流逻辑,并将特定事件发送到工作微服务以告诉他们要做什么。
In the orchestration pattern a central microservice, the orchestrator, issues commands to and awaits responses from subordinate worker microservices. Orchestration can be thought of much like a musical orchestra, where a single conductor commands the musicians during the performance. The orchestrator microservice contains the entire workflow logic for a given business process and sends specific events to worker microservices to tell them what to do.
编排器等待来自所指示的微服务的响应,并根据工作流逻辑处理结果。这与精心设计的工作流程形成鲜明对比,其中没有集中协调。
The orchestrator awaits responses from the instructed microservices and handles the results according to the workflow logic. This is in contrast to the choreographed workflow, in which there is no centralized coordination.
编排模式允许在单个微服务内灵活定义工作流程。编排器跟踪工作流程的哪些部分已完成、哪些部分正在进行中以及哪些部分尚未开始。工作流编排器向从属微服务发出命令事件,这些微服务通常通过事件流发出请求和响应来执行所需的任务并将结果提供回编排器。
The orchestration pattern allows for a flexible definition of the workflow within a single microservice. The orchestrator keeps track of which parts of the workflow have been completed, which are in process, and which have yet to be started. The workflow orchestrator issues command events to subordinate microservices, which perform the required task and provide the results back to the orchestrator, typically by issuing requests and responses through an event stream.
如果支付微服务在失败之前尝试完成支付三次,则它必须在支付微服务内部进行这三次尝试。它不会进行一次尝试并通知编排器失败并等待被告知是否重试。编排者不应该对支付的处理方式没有发言权,包括进行多少次尝试,因为这是支付微服务有界上下文的一部分。协调器唯一需要知道的是支付是否完全成功或完全失败。从那里,它可以根据工作流逻辑采取相应的行动。
If a payment microservice attempts to fulfill payment three times before failing, it must make those three attempts internal to the payment microservice. It does not make one attempt and notify the orchestrator that it failed and wait to be told to try again or not. The orchestrator should have no say about how payments are processed, including how many attempts to make, as that is part of the bounded context of the payment microservice. The only thing the orchestrator needs to know is if the payment has completely succeeded or if it has completely failed. From there, it may act accordingly based on the workflow logic.
确保编排器的有界上下文严格限于工作流逻辑,并且包含最少的业务履行逻辑。编排器仅包含工作流逻辑,而编排下的服务包含大部分业务逻辑。
Ensure the orchestrator’s bounded context is limited strictly to workflow logic and that it contains minimal business fulfillment logic. The orchestrator contains only the workflow logic, while the services under orchestration contain the bulk of the business logic.
请注意,编排模式中非编排器微服务的业务职责与编排模式中相同微服务的业务职责相同。编排器仅负责编排和工作流逻辑,根本不负责任何微服务本身的业务逻辑的实现。让我们看一个简单的例子来说明这些边界。
Note that the business responsibilities of a nonorchestrator microservice in an orchestrated pattern are identical to those of the same microservice in the choreographed pattern. The orchestrator is responsible only for orchestration and workflow logic, and not at all for the fulfillment of business logic of any of the microservices themselves. Let’s look at a simple example that illustrates these boundaries.
Figure 8-3 shows an orchestration version of the architecture in Figure 8-1.
编排器保留发送到服务 A、B 和 C 的事件的具体化,并根据工作微服务返回的结果更新其内部状态(参见表 8-1 )。
The orchestrator keeps a materialization of the events issued to services A, B, and C, and updates its internal state based on the results returned from the worker microservice (see Table 8-1).
| 输入事件ID | 服务A | 服务B | 服务C | 地位 |
|---|---|---|---|---|
100 100 |
<结果> <results> |
<结果> <results> |
<结果> <results> |
完毕 Done |
101 101 |
<结果> <results> |
<结果> <results> |
已派遣 Dispatched |
加工 Processing |
102 102 |
已派遣 Dispatched |
无效的 null |
无效的 null |
加工 Processing |
事件 ID 100 已成功处理,而事件 ID 101 和 102 处于工作流的不同阶段。编排器可以根据这些结果做出决策,并根据工作流逻辑选择下一步。处理事件后,协调器还可以从服务 A、B 和 C 的结果中获取必要的数据并组成最终输出。假设服务 A、B 和 C 中的操作彼此独立,您只需更改事件发送的顺序即可更改工作流。在以下编排代码中,事件只是从每个输入流中消费并根据工作流业务逻辑进行处理:
Event ID 100 has been successfully processed, while event IDs 101 and 102 are in different stages of the workflow. The orchestrator can make decisions based on these results and select the next step according to the workflow logic. Once the events are processed, the orchestrator can also take the necessary data from service A, B, and C’s results and compose the final output. Assuming the operations in services A, B, and C are independent of one another, you can make changes to the workflow simply by changing the order in which events are sent. In the following orchestration code, events are simply consumed from each input stream and processed according to the workflow business logic:
while(true){Event[]events=consumer.consume(streams)for(Eventevent:events){if(event.source=="Input Stream"){//process event + update materialized stateproducer.send("Stream 1",...)//Send data to stream 1}elseif(event.source=="Stream 1-Response"){//process event + update materialized stateproducer.send("Stream 2",...)//Send data to stream 2}elseif(event.source=="Stream 2-Response"){//process event + update materialized stateproducer.send("Stream 3",...)//Send data to stream 3}elseif(event.source=="Stream 3-Response"){//process event, update materialized state, and build outputproducer.send("Output",...)//Send results to output}}consumer.commitOffsets()}
while(true){Event[]events=consumer.consume(streams)for(Eventevent:events){if(event.source=="Input Stream"){//process event + update materialized stateproducer.send("Stream 1",...)//Send data to stream 1}elseif(event.source=="Stream 1-Response"){//process event + update materialized stateproducer.send("Stream 2",...)//Send data to stream 2}elseif(event.source=="Stream 2-Response"){//process event + update materialized stateproducer.send("Stream 3",...)//Send data to stream 3}elseif(event.source=="Stream 3-Response"){//process event, update materialized state, and build outputproducer.send("Output",...)//Send results to output}}consumer.commitOffsets()}
编排还可以使用请求-响应模式,其中编排器同步调用微服务的 API 并等待结果响应。除了直接调用的替代之外,图 8-4中所示的拓扑与图 8-3中的拓扑几乎相同。
Orchestration can also use a request-response pattern, where the orchestrator synchronously calls the microservice’s API and awaits a response for results. The topology shown in Figure 8-4 is nearly identical to the one in Figure 8-3, aside from substitution of direct calls.
直接呼叫服务的正常优点和限制也适用于此。话虽这么说,这种模式对于使用功能即服务解决方案实现工作流特别有用(请参阅第 9 章)。
The normal benefits and restrictions of direct-call services apply here as well. That being said, this pattern is particularly useful for implementing workflows with Function-as-a-Service solutions (see Chapter 9).
仔细观察时,直接调用和事件驱动的编排工作流程非常相似。例如,事件驱动系统似乎实际上只是一个请求-响应系统,在这些简单的示例中,它确实如此。但是,当您缩小一点时,在选择使用哪个选项时需要考虑许多因素。
Direct-call and event-driven orchestration workflows are fairly similar when examined close-up. For instance, it might seem that the event-driven system is really just a request-response system, and in these simple examples, it certainly is. But when you zoom out a bit, there are a number of factors to consider when choosing which option to use.
事件驱动的工作流程:
Event-driven workflows:
可以使用与其他事件驱动微服务相同的 I/O 监控工具和滞后扩展功能
Can use the same I/O monitoring tooling and lag scaling functionality as other event-driven microservices
允许事件流仍由其他服务使用,包括编排之外的服务
Allow event streams to still be consumed by other services, including those outside of the orchestration
通常更持久,因为协调器和依赖服务都通过事件代理与彼此的间歇性故障隔离
Are generally more durable, as both the orchestrator and the dependent services are isolated from each other’s intermittent failures via the event broker
具有内置的失败重试机制,因为事件可以保留在事件流中以便重试
Have a built-in retry mechanism for failures, as events can remain in the event stream for retrying
直接呼叫工作流程:
Direct-call workflows:
通常更快,因为事件流的生成和消费没有开销
Are generally faster, as there’s no overhead in producing to and consuming from event streams
存在可能需要由协调器管理的间歇性连接问题
Have intermittent connectivity issues that may need to be managed by the orchestrator
总而言之,如果所有依赖服务都在其 SLA 范围内运行,直接调用(又称为同步请求响应工作流程)通常比事件驱动的工作流程更快。当需要非常快速的响应时,例如在实时操作中,它们往往效果很好。同时,事件驱动的工作流程具有更持久的 I/O 流,产生更慢但更稳健的操作,并且特别擅长处理间歇性故障。
To summarize, direct-call aka synchronous request-response workflows are generally faster than event-driven ones, provided all dependent services are operating within their SLAs. They tend to work well when very quick responses are required, such as in real-time operations. Meanwhile, event-driven workflows have more durable I/O streams, producing slower but more robust operations, and are particularly good at handling intermittent failures.
请记住,有很多机会混合和匹配这两个选项。例如,编排工作流程可能主要是事件驱动的,但要求直接对外部 API 或预先存在的服务进行请求-响应调用。将这两个选项混合在一起时,请确保按预期处理每个服务的故障模式。
Keep in mind that there’s quite a lot of opportunity to mix and match these two options. For example, an orchestration workflow may be predominantly event-driven, but require that a request-response call be made directly to an external API or preexisting service. When mixing these two options together, make sure that each service’s failure modes are handled as expected.
编排器可以通过具体化每个传入和传出事件流以及响应请求结果来跟踪工作流中的事件。工作流本身仅在编排服务中定义,允许通过单点更改来更改工作流。在许多情况下,您可以将更改合并到工作流程中,而不会中断部分处理的事件。
The orchestrator can keep track of events in the workflow by materializing each of the incoming and outgoing event streams and response-request results. The workflow itself is defined solely within the orchestration service, allowing a single point of change for altering the workflow. In many cases, you can incorporate changes to a workflow without disrupting partially processed events.
编排导致服务之间的紧密耦合。必须显式定义协调器和依赖的工作人员服务之间的关系。
Orchestration results in a tight coupling between services. The relationship between the orchestrator and the dependent worker services must be explicitly defined.
确保编排器仅负责编排业务工作流程非常重要。一个常见的反模式是创建一个单一的“上帝”服务,向许多弱的 Minion 服务发出细粒度的命令。这种反模式在编排器和工作人员服务之间传播工作流业务逻辑,导致封装不良、定义不明确的有界上下文,并且难以将工作流微服务的所有权扩展到单个团队之外。编排器应将全部责任委托给依赖的工作人员服务,以最大程度地减少其执行的业务逻辑量。
It is important to ensure that the orchestrator is responsible only for orchestrating the business workflow. A common anti-pattern is creating a single “God” service that issues granular commands to many weak minion services. This anti-pattern spreads workflow business logic between the orchestrator and the worker services, making for poor encapsulation, ill-defined bounded contexts, and difficulty in scaling ownership of the workflow microservices beyond a single team. The orchestrator should delegate full responsibility to the dependent worker services to minimize the amount of business logic it performs.
您可以通过查询具体化状态来了解编排工作流程,因此可以轻松查看任何特定事件的进度以及工作流程中可能出现的任何问题。您可以在协调器级别实施监控和日志记录,以检测导致工作流失败的任何事件。
You gain visibility into the orchestration workflow by querying the materialized state, so it’s easy to see the progress of any particular event and any issues that may have arisen in the workflow. You can implement monitoring and logging at the orchestrator level to detect any events that result in workflow failures.
分布式事务是跨越两个或多个微服务的事务。每个微服务负责处理其事务部分,并在事务中止和恢复时反转该处理。履行和逆转逻辑必须驻留在同一个微服务中,既是出于可维护性的目的,也是为了确保新事务在无法回滚的情况下无法启动。
A distributed transaction is a transaction that spans two or more microservices. Each microservice is responsible for processing its portion of the transaction, as well as reversing that processing in the case that the transaction is aborted and reverted. Both the fulfillment and reversal logic must reside within the same microservice, both for maintainability purposes and to ensure that new transactions cannot be started if they can’t also be rolled back.
最好尽可能避免实施分布式事务,因为它们会给工作流程增加重大风险和复杂性。您必须考虑一系列问题,例如同步系统之间的工作、促进回滚、管理实例的瞬时故障以及网络连接等等。
It is best to avoid implementing distributed transactions whenever possible, as they can add significant risk and complexity to a workflow. You must account for a whole host of concerns, such as synchronizing work between systems, facilitating rollbacks, managing transient failures of instances, and network connectivity, to name just a few.
虽然最好尽可能避免实现分布式事务,但它们仍然有其用途,并且在某些情况下可能是必需的,特别是当这种避免会导致更大的风险和复杂性时。
While it is still best to avoid implementing distributed transactions whenever possible, they still have their uses and may be required in some circumstances, particularly when such avoidance would otherwise result in even more risk and complexity.
事件驱动世界中的分布式事务通常称为传奇,可以通过精心设计的模式或编排器模式来实现。saga 模式要求参与的微服务能够处理撤销操作以恢复其事务部分。常规处理操作和恢复操作都应该是幂等的,这样参与微服务的任何间歇性故障都不会导致系统处于不一致状态。
Distributed transactions in an event-driven world are often known as sagas and can be implemented through either a choreographed pattern or an orchestrator pattern. The saga pattern requires that the participating microservices be able to process reversal actions to revert their portion of the transaction. Both the regular processing actions and the reverting actions should be idempotent, such that any intermittent failures of the participating microservices do not leave the system in an inconsistent state.
具有精心设计的工作流程的分布式事务可能是复杂的事务,因为每个服务都需要能够在发生故障时回滚更改。这会在原本松散耦合的服务之间产生强耦合,并可能导致一些不相关的服务彼此严格依赖。
Distributed transactions with choreographed workflows can be complex affairs, as each service needs to be able to roll back changes in the event of a failure. This creates strong coupling between otherwise loosely coupled services and can result in some unrelated services having strict dependencies on one another.
精心设计的传奇模式适用于简单的分布式事务,特别是那些具有强烈的工作流排序要求且不太可能随时间变化的事务。监视精心设计的事务的进度可能很困难,因为它需要每个参与事件流的完全具体化,就像在精心设计的方法中一样。
The choreographed saga pattern is suitable for simple distributed transactions, particularly those with strong workflow ordering requirements that are unlikely to change over time. Monitoring the progress of a choreographed transaction can be difficult, because it requires a full materialization of each participating event stream, as in the orchestrated approach.
继续前面精心设计的工作流示例,考虑一系列微服务 A、B、C。服务 A 的输入事件流启动事务,服务 A、B 和 C 的工作完全完成,或者因此被取消并回滚。链中任何步骤的失败都会中止事务并开始回滚。最终的工作流程如图 8-5所示。
Continuing with the previous choreographed workflow example, consider the series of microservices A, B, C. The input event stream to service A kicks off a transaction, with the work of services A, B, and C being fully completed, or consequently canceled and rolled back. A failure at any step in the chain aborts the transaction and begins the rollback. The resultant workflow is shown in Figure 8-5.
现在假设服务 C 无法完成其事务部分。它现在必须通过发出事件或响应先前服务的请求来反转工作流程。服务 B 和 A 必须按顺序恢复其事务部分,如图8-6所示。
Suppose now that service C is unable to complete its part of the transaction. It must now reverse the workflow, either by issuing events or by responding to the previous service’s request. Services B and A must revert their portion of the transaction, in order, as shown in Figure 8-6.
服务 A(输入事件的原始使用者)现在必须决定如何处理失败的事务结果。前两张图中已经明显出现了一个奇怪的情况。成功交易的状态来自微服务 C 的输出。然而,中止交易的状态来自微服务 A,因此消费者需要监听 C 的输出和来自 A 的失败交易流才能获取已完成交易的完整情况。
Service A, the original consumer of the input event, must now decide what to do with the failed transaction results. A curious situation is already evident in the preceding two figures. The status of a successful transaction comes from the output of microservice C. However, the status of the aborted transaction comes out of microservice A, so a consumer would need to listen to both the output of C and the failed transaction stream from A to get a complete picture of finalized transactions.
记住单作者原则。不应将超过一项服务发布到事件流。
Remember the single-writer principle. No more than one service should publish to an event stream.
即使从输出和失败的交易流中进行消费,消费者仍然无法获取正在进行的交易或陷入处理过程中的交易的状态。这需要具体化每个事件流,或者通过 API 公开每个微服务的内部状态,如本章前面所述。对精心设计的事务的工作流进行更改需要处理与非事务性工作流相同的挑战,但会增加回滚工作流中先前微服务所做的更改的开销。
Even when consuming from both the output and failed transactions streams, the consumer will still not be able to get the status of ongoing transactions or of transactions that have gotten stuck in processing. This would require that each event stream be materialized, or that the internal state of each microservice be exposed via API, as discussed earlier in this chapter. Making changes to the workflow of a choreographed transaction requires dealing with the same challenges as a nontransactional workflow, but with the added overhead of rolling back changes made by the previous microservice in the workflow.
精心设计的交易可能有些脆弱,通常需要严格的排序,并且在监控方面可能存在问题。它们最适合具有极少数微服务的服务,例如具有非常严格的排序且需要工作流更改的可能性较低的一对或三个微服务。
Choreographed transactions can be somewhat brittle, generally require a strict ordering, and can be problematic to monitor. They work best in services with a very small number of microservices, such as a pair or a trio with very strict ordering and a low likelihood of needing workflow changes.
精心安排的事务建立在协调器模型的基础上,并添加了从工作流中的任何点恢复事务的逻辑。您可以通过反转工作流逻辑并确保每个工作微服务可以提供补充的反转操作来回滚这些事务。
Orchestrated transactions build on the orchestrator model, with the addition of logic to revert the transaction from any point in the workflow. You can roll back these transactions by reversing the workflow logic and ensuring that each worker microservice can provide a complementary reversing action.
精心安排的事务还可以支持各种信号,例如超时和人工输入。您可以使用超时来定期检查本地物化状态,以了解事务处理的时间。通过 REST API(参见第 13 章)的人工输入可以与其他事件一起处理,根据需要处理取消指令。协调器的集中式性质允许密切监控任何给定事务的进度和状态。
Orchestrated transactions can also support a variety of signals, such as timeouts and human inputs. You can use timeouts to periodically check the local materialized state to see how long a transaction has been processing. Human inputs via a REST API (see Chapter 13) can be processed alongside other events, handling cancellation instructions as required. The centralized nature of the orchestrator allows for close monitoring of the progress and state of any given transaction.
由于工作微服务之一的返回值、超时或人工操作员发送的中断,事务可能在工作流中的任何点中止。
The transaction can be aborted at any point in the workflow due to a return value from one of the worker microservices, a timeout, or an interrupt sent from a human operator.
A simple two-stage orchestrated transaction topology is shown in Figure 8-7.
事件从输入流中消耗并由编排器处理。在此示例中,编排器使用对工作流微服务的直接请求-响应调用。向服务 A 发出请求,协调器在等待响应时阻塞。一旦获得响应,编排器就会更新其内部状态并调用服务B,如图8-8所示。
Events are consumed from the input stream and processed by the orchestrator. In this example, the orchestrator is using direct request-response calls to the workflow’s microservices. A request is made to service A, and the orchestrator blocks while awaiting a response. Once it obtains the response, the orchestrator updates its internal state and calls service B, as shown in Figure 8-8.
服务 B 无法执行必要的操作,并且在用尽自己的重试和错误处理逻辑后,最终向协调器返回失败响应。编排器现在必须根据该事件的当前状态制定回滚逻辑,确保它向所有所需的微服务发出回滚命令。
Service B cannot perform the necessary operation and, after exhausting its own retries and error-handling logic, eventually returns a failure response to the orchestrator. The orchestrator must now enact its rollback logic based on the current state of that event, ensuring that it issues rollback commands to all required microservices.
每个微服务完全负责确保自己的重试策略、错误处理和间歇性故障管理。协调器不管理其中任何一个。
Each microservice is fully responsible for ensuring its own retry policy, error handling, and intermittent failure management. The orchestrator does not manage any of these.
图 8-9演示了协调器向服务 A 发出回滚命令(服务 B 的失败响应表明它没有向其内部数据存储写入任何内容)。在此示例中,服务 A 成功执行回滚,但如果在回滚期间失败,则将由编排器确定下一步要做什么。编排器可以多次重新发出命令,通过监控框架发出警报,或者终止应用程序以防止出现进一步的问题。
Figure 8-9 demonstrates the orchestrator issuing a rollback command to service A (service B’s failure response indicates it did not write anything to its internal data store). In this example, Service A performs the rollback successfully, but if it were to fail during its rollback, it would be up to the orchestrator to determine what to do next. The orchestrator could reissue the command a number of times, issue alerts via monitoring frameworks, or terminate the application to prevent further issues.
事务回滚后,协调器将决定下一步做什么以完成该事件的处理。它可能会多次重试该事件、丢弃该事件、终止应用程序或输出失败事件。协调器作为单一生产者,将事务失败发布到输出流,并让下游消费者处理它。这与编排不同,在编排中,没有单个流可以在不放弃单写入器原则的情况下消耗所有输出。
Once the transaction has been rolled back, it is up to the orchestrator to decide what to do next to finalize the processing of that event. It may retry the event a number of times, discard it, terminate the application, or output a failure event. The orchestrator, being the single producer, publishes the transaction failure to the output stream and lets a downstream consumer handle it. This differs from choreography, where there is no single stream from which to consume all output without discarding the single writer principle.
正如每个微服务完全负责自己的状态更改一样,它还负责确保回滚后其状态保持一致。在这种情况下,编排器的职责仅限于发出回滚命令并等待相关微服务的确认。
Just as each microservice is fully responsible for its own state changes, it is also responsible for ensuring that its state is consistent after a rollback. The orchestrator’s responsibility in this scenario is limited to issuing the rollback commands and awaiting confirmations from the dependent microservices.
编排器还可以在同一输出流中公开正在进行的事务的状态,在工作服务返回结果时更新事务实体。这可以提供对底层事务状态的高度可见性,并允许基于流的监控。
The orchestrator can also expose the status of transactions in progress in the same output stream, updating the transaction entity as worker services return results. This can provide high visibility into the state of underlying transactions and allows for stream-based monitoring.
与精心设计的事务相比,精心安排的事务可以更好地了解工作流依赖性、更大的更改灵活性以及更清晰的监控选项。编排器实例增加了工作流的开销并需要 管理,但可以为复杂的工作流提供精心设计的事务无法提供的清晰度和结构。
Orchestrated transactions offer better visibility into workflow dependencies, more flexibility for changes, and clearer monitoring options than choreographed transactions. The orchestrator instance adds overhead to the workflow and requires management, but can provide to complex workflows the clarity and structure that choreographed transactions cannot provide.
并非所有工作流程都需要完全可逆并受事务约束。给定的工作流程中可能会出现许多不可预见的问题,在许多情况下,您可能只需要尽力完成它即可。如果失败,您可以在事后采取一些措施来纠正这种情况。
Not all workflows need to be perfectly reversible and constrained by transactions. There are many unforeseen issues that can arise in a given workflow, and in many cases you might just have to do your best to complete it. In case of failure, there are actions you can take after the fact to remedy the situation.
基于票务和库存的系统经常使用这种方法。例如,销售实物产品的网站在购买时可能没有足够的库存来处理大量并发交易。在结算付款并评估其可用库存后,零售商可能会发现没有足够的库存来履行订单。此时它有多种选择。
Ticketing and inventory-based systems often use this approach. For example, a website that sells physical products may not have sufficient inventory at the time of purchase to handle a number of concurrent transactions. Upon settling the payments and evaluating its available inventory, the retailer may discover that there is insufficient stock to fulfill the orders. It has several options at this point.
严格的基于交易的方法将要求回滚最近的交易,即,将资金退还给支付提供商,并提醒客户该商品现在缺货并且订单已被取消。虽然技术上是正确的,但这可能会导致糟糕的客户体验以及客户和零售商之间缺乏信任。补偿工作流程可以根据企业的客户满意度政策来纠正这种情况。
A strict transaction-based approach would require that the most recent transactions be rolled back—that is, the money returned to the payment provider, and the customer alerted that the item is now out of stock and the order has been cancelled. While technically correct, this could lead to a poor customer experience and a lack of trust between the customer and the retailer. A compensating workflow can remedy the situation based on the business’s customer satisfaction policies.
作为一种补偿形式,企业可以订购新库存,通知客户出现了延误,并为下次购买提供折扣代码以示歉意。客户可以选择取消订单或等待新库存到达。音乐、体育和其他表演场所在门票超售的情况下经常使用这种方法,航空公司和其他以门票为基础的旅行社也是如此。补偿工作流并不总是可行,但它们通常对于处理面向客户的产品的分布式工作流操作很有用。
As a form of compensation, the business could order new stock, notify the customer that there has been a delay, and offer a discount code for the next purchase as an apology. The customer could be given the option to cancel the order or wait for the new stock to arrive. Music, sport, and other performance venues often use this approach in the case of oversold tickets, as do airlines and other ticket-based travel agencies. Compensation workflows are not always possible, but they are often useful for handling distributed workflow operations with customer-facing products.
编排允许业务部门和独立工作流程之间的松散耦合。它适用于简单的分布式事务和简单的非事务性工作流程,其中微服务数量较少且业务操作的顺序不太可能改变。
Choreography allows for loose coupling between business units and independent workflows. It is suitable for simple distributed transactions and simple nontransactional workflows, where the microservice count is low and the order of business operations is unlikely to ever change.
与编排相比,精心编排的事务和工作流程可提供更好的工作流程可见性和监控。它们可以处理比编排更复杂的分布式事务,并且通常可以在单个位置进行修改。容易发生变化并包含许多独立微服务的工作流非常适合编排器模式。
Orchestrated transactions and workflows provide better visibility and monitoring into workflows than choreography. They can handle more complicated distributed transactions than choreography and can often be modified in just a single location. Workflows that are subject to changes and contain many independent microservices are well suited to the orchestrator pattern.
最后,并非所有系统都需要分布式事务才能成功运行。某些工作流程可以在发生故障时提供补偿操作,依靠业务的非技术部分来解决客户面临的问题。
Finally, not all systems require distributed transactions to successfully operate. Some workflows can provide compensatory actions in the case of failure, relying on nontechnical parts of the business to solve customer-facing issues.
功能即服务(FaaS)是一种“无服务器”解决方案,近年来变得越来越流行。FaaS 解决方案使个人能够构建、管理、部署和扩展应用程序功能,而无需管理基础设施开销。它们可以在事件驱动系统中提供重要价值,作为实现简单到中等复杂解决方案的一种手段。
Functions-as-a-Service (FaaS) is a “serverless” solution that has become increasingly popular in recent years. FaaS solutions enable individuals to build, manage, deploy, and scale application functionality without having to manage infrastructural overhead. They can provide significant value in event-driven systems as a means of implementing simple to moderately complex solutions.
函数是当特定触发条件发生时执行的一段代码。该函数启动,运行直至完成,然后在其工作完成后终止。FaaS 解决方案可以根据负载轻松地增加或减少函数执行的数量,从而为高度可变的负载提供密切跟踪。
A function is a piece of code that is executed when a specific triggering condition occurs. The function starts up, runs until completion, and then terminates once its work is completed. FaaS solutions can easily scale the number of function executions up and down depending on load, providing close tracking for highly variable loads.
将 FaaS 解决方案视为经常失败的基本消费者/生产者实现可能会有所帮助。函数总是会在预定的时间后结束,并且与其关联的任何连接和状态都将消失。在设计函数时请记住这一点。
It may be helpful to think of a FaaS solution as a basic consumer/producer implementation that regularly fails. A function will always end after a predetermined amount of time, and any connections and state associated with it will go away. Keep this in mind as you design your functions.
FaaS 解决方案可能包含许多不同的功能,它们的操作总和构成了业务边界上下文的解决方案。创建基于功能的解决方案的方法有很多,远远超出了本章所涵盖的范围,但有一些通用设计原则将帮助指导您完成整个过程。
FaaS solutions may comprise many different functions, with the sum of their operations constituting the solution to the business bounded context. There are many ways to create function-based solutions, far more than this chapter can cover, but there are a few general design principles that will help guide you through the process.
组成解决方案的函数和内部事件流必须严格属于有界上下文,以便清楚地识别函数和数据的所有者。在大量实施微服务解决方案时,组织通常会遇到有关功能、服务和事件流的所有权问题 。虽然许多微服务解决方案以 1:1 的方式映射到有界上下文,但n :1 映射并不罕见,因为单个有界上下文可以使用多个函数。确定哪个函数属于哪个有界上下文非常重要,因为函数的高粒度可能会模糊这些界限。
The functions and internal event streams composing a solution must strictly belong to a bounded context, such that the owner of the function and data is clearly identified. It is common for an organization to have ownership questions around functions, services, and event streams when implementing microservice solutions in large numbers. While many microservice solutions map 1:1 to a bounded context, an n:1 mapping is not uncommon, as multiple functions may be used for a single bounded context. It’s important to identify which function belongs to which bounded context, because the high granularity of functions can blur those lines.
使用函数维护有界上下文的一些实用方法包括:
Some practical ways of maintaining bounded contexts with functions include:
确保数据存储对外部上下文保密。
Ensure that data stores are kept private from external contexts.
耦合到其他上下文时,使用标准请求响应或事件驱动接口。
Use standard request-response or event-driven interfaces when coupling to other contexts.
围绕哪个功能属于哪个上下文维护严格的元数据(功能到产品的 1:1 映射)。
Maintain strict metadata around which function belongs to which context (a 1:1 mapping of function to product).
在映射到有界上下文的存储库中维护函数代码。
Maintain function code within a repository mapped to the bounded context.
偏移量在以下两个时间之一提交:函数启动时或函数完成处理时。仅在给定事件或一批事件的处理完成后提交偏移量是 FaaS 最佳实践。了解如何处理基于函数的特定解决方案的偏移对您来说非常重要,因此让我们看一下每种方法的含义。
Offsets are committed at one of two times: when the function starts or when the function has completed its processing. Committing offsets only after processing has completed for a given event or batch of events is a FaaS best practice. It is important for you to understand how offsets for your specific function-based solutions are handled, so let’s take a look at the implications of each approach.
处理完成后提交偏移量的方法与其他微服务实现提交偏移量的方式一致,无论它们是基于基本的生产者/消费者还是流处理框架。该策略提供了事件至少被处理一次的最强保证,相当于非 FaaS 解决方案中使用的偏移量管理策略。
The approach of committing offsets after processing has completed aligns with how other microservice implementations commit offsets, whether they’re based on a basic producer/consumer or stream-processing framework. This strategy provides the strongest guarantee that events will be processed at least once and is equivalent to the offset management strategies used with non-FaaS solutions.
一旦要处理的一批事件被传递给函数,就可以提交偏移量。这种简单的方法被用在许多 FaaS 框架中,这些框架依赖于特定于框架的重试机制和警报来减轻重复的事件处理失败。以编排模式调用其他函数的函数通常依赖于此策略,因为它极大地简化了事件处理的跟踪。
Offsets may be committed once the batch of events to process has been passed off to the function. This simple approach is used in many FaaS frameworks that rely on framework-specific retry mechanisms and alerting to mitigate repetitive event processing failures. Functions that call other functions in a choreography pattern often rely on this strategy, as it greatly simplifies the tracking of event processing.
然而,在处理完成之前提交偏移量可能会出现问题。如果该函数无法成功处理事件,并且多次重试失败,则可能会丢失数据。该事件通常会被分流到死信队列中或简单地丢弃。虽然许多基于功能的微服务对数据丢失不敏感,但那些对数据丢失敏感的微服务不应使用此策略。
Committing offsets before processing has completed can be problematic, however. If the function is unable to successfully process an event, and numerous retries fail, then data loss is likely. The event will typically be shunted into a dead-letter queue or simply discarded. While many function-based microservices are not sensitive to data loss, those that are should not use this strategy.
FaaS 框架的一个经常被引用的功能是它们可以轻松编写单个函数并在多个服务中重用它。然而,采用这种方法可能会导致解决方案高度分散,从而难以准确辨别有界上下文中发生的情况。此外,给定功能的所有权变得不明确,并且可能不清楚功能的更改是否会对其他服务产生负面影响。虽然功能的版本控制可以帮助解决这个问题,但当多个产品需要维护和改进功能的不同版本时,它也可能导致冲突。
An often-cited feature of FaaS frameworks is that they make it easy to write a single function and reuse it in multiple services. Following this approach, however, can lead to a highly fragmented solution that makes it difficult to discern exactly what is going on within the bounded context. Additionally, ownership of a given function becomes ambiguous, and it may not be clear if changes to a function could negatively affect other services. While versioning of functions can help with this issue, it can also lead to conflicts when multiple products need to maintain and improve different versions of a function.
FaaS 解决方案可能会合并多个函数来解决有界上下文的业务需求,虽然这并不是一种罕见或不好的做法,但 FaaS 解决方案的一个好的经验法则是,更少的函数比许多细粒度的函数更好。测试、调试和管理一个功能比对多个功能进行相同的操作要容易得多。
FaaS solutions may incorporate multiple functions to solve the business requirements of the bounded context, and while this is not an uncommon or bad practice, a good rule of thumb for FaaS solutions is that fewer functions are better than many granular functions. Testing, debugging, and managing just one function is much easier than doing the same for multiple functions.
就像事件代理和容器管理系统 (CMS) 一样,FaaS 框架既可以作为免费使用的开源解决方案,也可以作为付费的第三方云提供商提供。为其微服务运营运行自己的内部 CMS 的组织也可以从使用内部 FaaS 解决方案中受益。有许多免费的开源选项可用,例如 OpenWhisk、OpenFaaS 和 Kubeless,它们可以利用现有的容器管理服务。Apache Pulsar 提供自己的内置 FaaS 解决方案,与其事件代理一起运行。通过利用通用的资源配置框架,您的 FaaS 解决方案可以与您的微服务解决方案保持一致。
Just like the event brokers and container management systems (CMSes), FaaS frameworks are available as both free-to-use open source solutions and paid third-party cloud providers. An organization that runs its own in-house CMS for its microservice operations can also benefit from using an in-house FaaS solution. There are a number of free open source options available, such as OpenWhisk, OpenFaaS, and Kubeless, that can leverage existing container management services. Apache Pulsar offers its own built-in FaaS solution that runs alongside its event broker. By leveraging a common resource-provisioning framework, your FaaS solution can align with your microservice solution.
Amazon Web Services (AWS)、Google Cloud Platform (GCP) 和 Microsoft Azure 等第三方服务提供商也拥有自己专有的 FaaS 框架,每个框架都提供有吸引力的特性和功能,但仍与其提供商的专有事件代理紧密集成。这是一个重要问题,因为目前所有三个提供商都将其事件代理内的保留时间限制为 7 天。云提供商和开源事件代理之间的集成确实存在(例如Kafka Connect),但可能需要额外的工作来设置和管理。话虽如此,如果您的组织已经是 AWS、GCP 或 Azure 服务的订阅者,那么开始试验的开销就会很低。
Third-party service providers such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure also have their own proprietary FaaS frameworks, each of which offers attractive features and functionality but remains tightly integrated with its provider’s proprietary event broker. This is a significant issue only because currently all three providers limit retention within their event brokers to seven days. Integrations between cloud providers and open source event brokers do exist (such as Kafka Connect), but may require additional effort to set up and manage. That being said, if your organization is already a subscriber to AWS, GCP, or Azure services, then the overhead to start experimenting is low.
无论您的 FaaS 框架或事件代理如何,在使用基于函数的解决方案时都必须考虑四个主要组件:
There are four main components you must consider when working with function-based solutions, regardless of your FaaS framework or event broker:
功能
The function
输入事件流
Input event stream
触发逻辑
Triggering logic
错误和扩展策略,以及元数据
Error and scaling policies, with metadata
FaaS 实现的第一个组件是函数本身。它可以在 FaaS 框架支持的任何代码中实现。
The first component of a FaaS implementation is the function itself. It can be implemented in any code supported by the FaaS framework.
publicintmyfunction(Event[]events,Contextcontext){println("hello world!");return0;}
publicintmyfunction(Event[]events,Contextcontext){println("hello world!");return0;}
该events参数包含要处理的各个事件的数组,每个事件包含key、value、timestamp、offset和partition_id。该Context参数包含有关函数及其上下文的信息,例如其名称、事件流 ID 和函数的剩余生命周期。
The events parameter contains the array of individual events to be processed, with each event containing a key, value, timestamp, offset, and partition_id. The Context parameter contains information about the function and its context, such as its name, the event stream ID, and the function’s remaining lifespan.
接下来,您需要为该函数连接一些触发逻辑。这将在下一节中更详细地介绍,但现在假设每当新事件到达其订阅的事件流之一时就会触发该函数。触发逻辑通常通过函数触发映射与函数关联,该映射通常隐藏在 FaaS 框架的幕后。这是一个例子:
Next, you need to wire up some triggering logic for the function. This will be covered in greater detail in the next section, but for now, say that the function is triggered whenever a new event arrives in one of its subscribed event streams. The triggering logic is often associated with a function by way of a function-trigger map, which is usually concealed behind the scenes of your FaaS framework. Here is an example:
| 功能 | 事件流 | 扳机 | 政策和元数据 |
|---|---|---|---|
|
|
|
<…> < … > |
您可以看到它myFunction被设置为在新事件传递到 时触发myInputStream。您还会注意到,有一个名为“策略和元数据”的列,该列有点包罗万象,其中包括如下配置:
You can see that myFunction is set to trigger when a new event is delivered to myInputStream. You’ll also notice that there is a column named Policies and Metadata, which is a bit of a catch all that includes configurations such as the following:
消费群体
Consumer group
消费者属性,例如批量大小和批量窗口
Consumer properties, such as batch size and batch window
重试和错误处理策略
Retry and error handling policies
扩展政策
Scaling policies
一旦建立了触发器、元数据和策略,该功能就准备好处理传入事件。当新事件到达其输入事件流时,FaaS 框架将启动该函数,获取一批事件并开始处理。完成后,该函数将终止并等待更多事件进入。这是事件流侦听器模式的典型实现,下一节将详细讨论。
Once the triggers, metadata, and policies are established, the function is ready to process incoming events. When a new event arrives in its input event stream, the function will be started by the FaaS framework, get passed a batch of events, and begin processing. Upon completion, the function will terminate and wait for more events to come in. This is a typical implementation of the event-stream listener pattern, which is discussed more in the next section.
每个基于功能的微服务实现都必须有自己独立的消费者组,就像其他非 FaaS 微服务一样。
Each function-based microservice implementation must have its own independent consumer group, just as with other non-FaaS microservices.
现在请记住,这只是成功触发和操作功能所需组件的逻辑表示。FaaS 框架的功能编码要求、功能管理和触发机制因提供商和实现而异,因此请务必参考您的 FaaS 框架的文档。
Now keep in mind that this is just a logical representation of the components needed to successfully trigger and operate a function. A FaaS framework’s function coding requirements, function management, and triggering mechanisms vary by provider and implementation, so be sure to refer to the documentation for your FaaS framework.
触发机制、事件消费、消费者偏移、嵌套函数、故障和至少一次事件处理之间也存在适度复杂的相互作用。这些是本章剩余部分的主题。
There is also a moderately complex interplay between triggering mechanisms, event consumption, consumer offsets, nested functions, failures, and at-least-once event processing. These are the subject of the remainder of this chapter.
冷启动是功能首次启动时或长时间不活动后的默认状态。必须启动容器并加载代码,可能需要创建事件代理连接,并且需要建立与外部资源的任何其他客户端连接。一旦一切稳定并准备好处理,该函数现在就处于热状态并准备好处理事件。热函数开始处理事件,并在到期或完成时暂停并进入休眠状态。
A cold start is the default state of the function upon starting for the first time, or after a sufficient period of inactivity. A container must be started and the code loaded, event broker connections may need to be created, and any other client connections to external resources need to be established. Once everything is stable and ready to process, the function is now in a warm state and ready to process events. The warm function begins processing events, and upon expiry or completion, is suspended and put into hibernation.
大多数 FaaS 框架都会尽可能重用已终止的函数。在许多情况下,处理稳定的事件流的函数会遇到超时到期并短暂终止,稍后会被触发机制恢复。挂起的实例将被简单地重用,并且只要与事件代理和任何状态存储的连接在此期间没有过期,处理就可以立即恢复。
Most FaaS frameworks attempt to reuse terminated functions whenever possible. In many scenarios, a function processing a steady stream of events will hit the timeout expiry and be briefly terminated, just to be brought back a moment later by a triggering mechanism. The suspended instance is simply reused, and provided that the connections to the event broker and any state stores haven’t expired during the interim, processing can resume immediately.
触发器用于告诉函数启动并开始处理。支持的触发器根据您的 FaaS 框架而有所不同,但往往都属于相同的一般类别,如稍后所述。现在,让我们看看哪些信号可用于启动函数,以便您了解何时需要使用它们。
Triggers are used to tell a function to start up and begin processing. The supported triggers vary depending on your FaaS framework, but tend to all fall into the same general categories, as described shortly. For now, let’s take a look at which signals can be used to kick off functions, to give you an idea of when you may want to use them.
当事件产生到事件流中时可以触发函数。事件流侦听器触发器将事件消耗隔离在预定义使用者后面,从而减少开发人员必须编写的开销代码量。事件以事件数组的形式直接注入到函数中,按照事件流的顺序排列,或者如果从队列中消费,则作为无序事件的集群。您可以创建从事件流到函数的多个映射,以便函数可以使用来自许多不同流的事件。
Functions can be triggered when an event is produced into an event stream. The event-stream listener trigger isolates event consumption behind a predefined consumer, reducing the amount of overhead code that a developer must write. Events are injected directly into the function in a form of an array of events, in sequential order from an event stream, or as a cluster of unordered events if consuming from a queue. You can create multiple mappings from event streams to functions, such that a function can consume events from many different streams.
Google、Microsoft 和 Amazon 的 FaaS 解决方案提供了此触发器,供其专有事件代理使用,但目前不支持直接从开源代理触发。该方法的一般结构如图 9-1所示。
FaaS solutions from Google, Microsoft, and Amazon provide this trigger for usage with their proprietary event brokers, but currently do not support triggering directly from open source brokers. The generalized structure of this approach is shown in Figure 9-1.
相反,OpenFaaS、Kubeless、Nuclio 等开源解决方案提供了各种带有各种事件代理的触发插件,例如 Kafka、Pulsar 和 NATS 等。例如,Apache Kafka Connect允许您触发第三方FaaS框架的功能。由于 Kafka Connect 在 FaaS 框架之外运行,因此您最终会得到如图 9-2所示的事件流侦听器。
Conversely, open source solutions such as OpenFaaS, Kubeless, Nuclio, and others provide a variety of triggering plug-ins with various event brokers, such as Kafka, Pulsar, and NATS, to name a few. For instance, Apache Kafka Connect allows you to trigger the functions of third-party FaaS frameworks. Since Kafka Connect runs outside of the FaaS framework, you would end up with an event-stream listener as per Figure 9-2.
尽管前面的示例中未显示,但函数结果可以输出到它们自己的事件流,不仅用于输出数据,还用于跟踪函数的成功。
Though not shown in the previous examples, function results can be output to their own event streams, not just for the purpose of outputting data but also for tracking the function’s success.
同步触发器要求函数在发出下一个事件之前完成。这对于维护处理顺序尤其重要,并且受到正在处理的事件流的并行性的限制。相反,异步触发可以向多个函数发出多个事件,每个函数在完成时都会报告。但是,这不会维护处理顺序,并且仅当处理顺序对业务逻辑不重要时才应使用。
Synchronous triggers require the function to complete before they issue the next events. This is particularly important for maintaining the processing order and is limited by the parallelism of the event stream being processed. Conversely, asynchronous triggering can issue multiple events to multiple functions, each one reporting back as it is completed. This will not maintain the processing order, however, and should be used only when processing order is not important to the business logic.
批处理大小和批处理窗口是流侦听器触发器中需要考虑的两个重要属性。批处理大小指示要分派处理的最大事件数,而批处理窗口指示等待其他事件(而不是立即触发函数)的最长时间。这两个参数都用于确保启动函数的开销分散在批量记录中,以降低成本。
Batch size and batch window are two important properties to consider in stream-listener triggers. The batch size dictates the maximum number of events to dispatch for processing, while the batch window indicates the maximum amount of time to wait for additional events, instead of triggering the function immediately. Both of these parameters are used to ensure that the overhead of starting the function is spread among the batch of records to reduce costs.
由流侦听器触发器执行的函数如下所示:
A function executed by a stream-listener trigger looks something like the following:
publicintmyEventfunction(Event[]events,Contextcontext){for(Eventevent:events)try{println(event.key+", "+event.value);}catch(Exceptione){println("error printing "+event.toString);}//Indicates to the FaaS framework that batch processing was completed.context.success();return0;}
publicintmyEventfunction(Event[]events,Contextcontext){for(Eventevent:events)try{println(event.key+", "+event.value);}catch(Exceptione){println("error printing "+event.toString);}//Indicates to the FaaS framework that batch processing was completed.context.success();return0;}
消费者组的滞后指标是触发函数的另一种方式。您可以通过定期轮询单个应用程序的消费者组的偏移量并计算当前消费者偏移量与流的头部偏移量之间的增量来检测延迟(有关延迟监控的更多信息,请参阅“消费者偏移延迟监控” )。与流侦听器触发器类似,滞后监控也可用于扩展非 FaaS 微服务。
A consumer group’s lag metric is another way to trigger functions. You can detect lag by periodically polling the offsets of an individual application’s consumer groups and computing the delta between the current consumer offset and the head offset of the stream (see “Consumer Offset Lag Monitoring” for more on lag monitoring). While similar to the stream listener trigger, lag monitoring can also be used for scaling non-FaaS microservices.
滞后监控通常涉及计算滞后指标并将其报告给您选择的监控框架。然后,监控框架可以调用FaaS框架来告诉它启动在事件流上注册的功能。高滞后值可以指示可以启动多个函数实例以更快地处理负载,而低滞后值可能只需要单个函数实例来处理积压。您可以逐个微服务定制滞后量和功能启动之间的关系,确保符合 SLA。
Lag monitoring typically involves computing and reporting lag metrics to your monitoring framework of choice. The monitoring framework can then call the FaaS framework to tell it to start up the functions registered on the event stream. A high lag value can indicate that multiple function instances can be started to more quickly process the load, while a low lag value may require only a single function instance to process the backlog. You can tailor the relationship between lag quantity and function startup on a microservice-by-microservice basis, ensuring compliance with SLAs.
前面提到的事件流侦听器触发器与这一事件流侦听器触发器之间的主要区别之一是,通过滞后触发,该函数直到启动后才会消耗事件。由滞后触发器启动的函数具有更广泛的职责范围,包括与事件代理建立客户端连接、使用事件以及提交回任何偏移更新。这使得由滞后触发的函数与基本的生产者/消费者客户端更加相似,尽管其生命周期有限。以下示例函数说明了此 工作流程:
One of the major differences between the previously mentioned event-stream listener trigger and this one is that with lag triggering, the function does not consume the events until after it is started. Functions started by the lag trigger have a wider domain of responsibilities, including establishing a client connection with the event broker, consuming the events, and committing back any offset updates. This makes functions triggered by lag much more similar to basic producer/consumer clients, albeit with a limited lifespan. The following example function illustrates this workflow:
publicintmyLagConsumerfunction(Contextcontext){StringconsumerGroup=context.consumerGroup;StringstreamName=context.streamName;EventBrokerClientclient=newEventBrokerClient(consumerGroup,...);Event[]events=client.consumeBatch(streamName,...);for(Eventevent:events){// Do event processing workdoWork(event);}//Commit the offsets back to the event brokerclient.commitOffsets();//Indicates to the FaaS framework that the function succeeded.context.success();//Return, letting the lag-triggering system know it was a successreturn0;}
publicintmyLagConsumerfunction(Contextcontext){StringconsumerGroup=context.consumerGroup;StringstreamName=context.streamName;EventBrokerClientclient=newEventBrokerClient(consumerGroup,...);Event[]events=client.consumeBatch(streamName,...);for(Eventevent:events){// Do event processing workdoWork(event);}//Commit the offsets back to the event brokerclient.commitOffsets();//Indicates to the FaaS framework that the function succeeded.context.success();//Return, letting the lag-triggering system know it was a successreturn0;}
消费者组和流名称作为上下文中的参数传递。创建客户端,使用和处理事件,并将偏移量提交回事件代理。该函数将success结果指示回FaaS框架,然后返回。
The consumer group and stream name are passed in as parameters in the context. The client is created, events are consumed and processed, and the offsets are committed back to the event broker. The function indicates a success result back to the FaaS framework and then returns.
如果该函数经常由滞后监视器触发,则很有可能它在上次迭代中仍然是热的,并且连接到事件代理客户端的开销可能不是一个因素。当然,这取决于客户端和事件代理配置使用的超时。对于较长时间的不活动,消费者组重新平衡和客户端冷启动将稍微减少函数实例可以处理的工作量。
If the function is frequently triggered by the lag monitor, there is a good chance that it will still be warm from the last iteration, and the overhead of connecting to the event broker client may not be a factor. This, of course, depends on the timeouts used by the client and event broker configurations. For longer periods of inactivity, consumer group rebalancing and client cold starts will slightly reduce the amount of work that a function instance can process.
还可以安排功能定期在特定日期时间启动。计划的函数以指定的时间间隔启动,轮询源事件流以查找新事件,并根据需要进行处理或关闭。轮询周期应保持较短,以便维持 SLA,但轮询过于频繁可能会给 FaaS 框架和事件代理带来过度的负载。
Functions can also be scheduled to start up periodically and at specific datetimes. The scheduled functions start up at the specified interval, poll the source event-streams for new events, and process them or shut down as necessary. The polling period should be kept low so that SLAs are maintained, but polling too frequently may put undue load on both the FaaS framework and event broker.
基于时间的触发器的客户端代码看起来与消费者组滞后触发器示例的客户端代码相同。
Client code for a time-based trigger looks identical to that of the consumer group lag trigger example.
对资源所做的更改也可能是触发因素的来源。例如,创建、更新或删除文件系统中的文件可以触发函数,就像对数据存储中的行进行相同的修改一样。由于事件驱动的微服务域中的大多数事件都是通过事件流生成的,因此这种特定的资源触发器在大多数业务工作流中并不经常使用。然而,当您与需要 FTP 或其他文件服务将文件放入其中的外部数据源集成时,它非常有用。
Changes made to resources can also be a source of triggers. For instance, creating, updating, or deleting a file in a filesystem can trigger functions, as can the same modifications made to a row in a data store. Since most of the events in the event-driven microservice domain are generated via event streams, this particular resource trigger isn’t often used in most business workflows. It is, however, quite useful when you are integrating with external sources of data that require an FTP or other file service to drop their files into.
FaaS 方法特别适合能够利用处理资源配置的按需、灵活特性的解决方案。简单的拓扑是一个很好的候选者,无状态的拓扑、不需要对多个事件流进行确定性处理的拓扑以及扩展范围非常广的拓扑(例如基于队列的处理)也是如此。任何具有高度可变容量的事物都可以从 FaaS 解决方案中受益,因为它们的水平扩展功能和按需特性允许快速配置和释放计算资源。
FaaS approaches work particularly well for solutions that can leverage the on-demand, flexible nature of processing resource provisioning. Simple topologies are a great candidate, as are those that are stateless, those that do not require deterministic processing of multiple event streams, and those that scale very wide, such as queue-based processing. Anything with a highly variable volume can benefit from FaaS solutions, as their horizontal scaling capabilities and on-demand characteristics allow for rapid provisioning and release of compute resources.
当不考虑并发性和确定性时,FaaS 解决方案可以表现得非常好。然而,一旦确定性发挥作用,您就必须非常小心地确保事件流处理的正确性和一致性。与下一章中的基本消费者解决方案非常相似,FaaS 解决方案要求您提供事件调度程序以确保一致的处理结果。共同分区的数据一次只能由单个函数成功且一致地处理,类似于全功能轻量级和重量级框架必须仅使用单个线程。
FaaS solutions can perform extremely well when concurrency and determinism are not concerns. However, once determinism comes into play, you must take great care to ensure correctness and consistency in the event stream processing. Much like the basic consumer solutions in the next chapter, FaaS solutions require that you provide an event scheduler to ensure consistent processing results. Copartitioned data can only be successfully and consistently processed by a single function at a time, similar to how the full-featured lightweight and heavyweight frameworks must use only a single thread.
鉴于功能的生命周期较短,大多数基于 FaaS 的有状态解决方案都需要使用外部有状态服务。部分原因是许多 FaaS 提供商的目标是提供独立于数据位置的快速、高度可扩展的处理能力单元。具有需要先前执行的本地状态的函数会将当前执行限制到具有该状态的同一位置的节点。这极大地降低了 FaaS 提供商的灵活性,因此他们经常强制执行“无本地状态”政策,并要求所有有状态的内容都存储在执行器外部。
Given the short lifespan of functions, most stateful FaaS-based solutions require the use of an external stateful service. Part of the reason is the goal of many FaaS providers to supply quick, highly scalable units of processing power independent of the data’s location. Having functions that require local state from previous executions limits current execution to the nodes that have that state co-located. This greatly reduces the flexibility of FaaS providers, so they often enforce a “no local state” policy and require that everything stateful be stored external to the executors.
尽管如果函数在热状态下启动,则先前的本地状态可能可用,但不能保证情况总是如此。函数与外部状态存储的连接与任何其他客户端完全相同 - 通过创建与状态存储的连接并使用相应的 API。任何状态都必须由函数显式地保存和检索。
Although previous local state may be available if a function starts in a warm state, there is no guarantee that this will always be the case. Functions connect to external state stores exactly as any other client would—by creating a connection to the state store and using the corresponding API. Any state must be persisted and retrieved explicitly by the function.
请务必对函数的状态使用严格的访问权限,以便不允许访问其有界上下文之外的任何内容。
Be sure to use strict access permissions for your function’s state, such that nothing outside of its bounded context is allowed access.
一些 FaaS 框架添加了持久有状态函数支持,例如Microsoft Azure 的 Durable Functions,它抽象了状态的显式管理,并允许您使用本地内存,该内存会自动保留到外部状态。这允许开发人员挂起函数并使它们恢复生机,而无需编写代码来显式存储和检索状态。这极大地简化了有状态工作流程,并提供了跨功能实现标准化状态管理的选项。
Some FaaS frameworks have added durable stateful function support, such as Microsoft Azure’s Durable Functions, which abstracts away the explicit management of state and allows you to use local memory, which is automatically persisted to external state. This allows developers to suspend functions and bring them back to life without needing to write code to explicitly store and retrieve the state. This greatly simplifies stateful workflows and provides the option to standardize state management across function implementations.
FaaS 框架将继续发展并包含新功能。简单的状态管理是基于函数的解决方案的常见需求,因此请留意您选择的 FaaS 框架中状态处理的改进。
FaaS frameworks will continue to grow and include new features. Simple management of state is a common need in function-based solutions, so keep a lookout for state handling improvements in the FaaS frameworks of your choice.
函数通常用于执行其他函数,也可用于精心设计和编排的工作流程。可以通过事件、请求-响应调用或两者的组合来异步促进函数之间的通信。这些选择很大程度上取决于 FaaS 框架和有界上下文的问题空间。在实现多功能解决方案时,通常使用编排和编排设计模式。
Functions are often used to execute other functions and may also be used for both choreographed and orchestrated workflows. Communication between functions can be facilitated asynchronously through events, via request-response calls, or with a combination of both. These choices depend highly on the FaaS framework and the problem space of the bounded context. It’s common to use choreography and orchestration design patterns when implementing a multifunction solution.
为了避免无序处理问题,请确保在处理下一个事件之前完成一个事件的所有处理。
To avoid out-of-order processing issues, ensure that all processing is completed for one event before processing the next event.
一个函数的输出可以生成到另一个消费函数的事件流中。有界上下文可以由许多函数和许多内部事件流组成,每个函数定义具有不同的触发和缩放逻辑。每个函数都以自己的速率处理传入事件,并相应地消耗事件、执行工作并产生输出。这种设计的一个例子如图 9-3所示。
The output of one function can be produced into an event stream for another consuming function. A bounded context may be made up of many functions and many internal event streams, with varying triggering and scaling logic for each function definition. Each function processes incoming events at its own rate, with events being consumed, work being performed, and outputs produced accordingly. An example of this design is shown in Figure 9-3.
在此示例中,功能 A 的触发独立于功能 B 和 C 的触发器。事件流 2 和 3 被视为内部事件流,对其内容的访问完全仅限于有界上下文之外的任何函数。每个函数都使用相同的使用者组来使用来自其源流的事件,因为这些函数都位于同一有界上下文中。这确保了这些功能以与非基于 FaaS 的微服务相同的方式有效运行。
In this example, function A is triggered independently of the triggers for functions B and C. Event streams 2 and 3 are considered internal event streams, with access to their contents completely restricted to any functions outside of the bounded context. Each function consumes events from its source stream using the same consumer group, as the functions are all colocated within the same bounded context. This ensures that the functions are effectively operating in the same way as a non-FaaS-based microservice.
使用基于事件的通信模式有几个好处。拓扑中的每个函数都可以管理自己的消费者偏移量,一旦其工作完成就提交每个偏移量。在事件流处理之外,功能之间不需要进行协调。上图显示了基于编排的设计模式,但也可以使用编排。此外,事件处理中的任何失败都不会导致任何数据丢失,因为事件会持久存储在事件代理中,并且只会由下一个函数实例重新处理。
There are several benefits of using an event-based communication pattern. Each function within the topology can manage its own consumer offsets, committing each offset once its work is done. No coordination between functions needs to occur outside of the event stream processing. The preceding figure shows a choreography-based design pattern, though orchestration can also be used. Additionally, any failures in the event processing will not result in any data loss, as the events are durably stored in the event broker and will simply be reprocessed by the next function instance.
在直接调用模式中,函数可以直接从其自己的代码中调用其他函数。其他函数的直接调用可以异步执行,这本质上是一种“即发即忘”的方法,也可以同步执行,其中调用函数等待返回值。
In the direct-call pattern, a function can call other functions directly from its own code. Direct invocation of other functions can be performed asynchronously, which is essentially a “fire-and-forget” approach, or synchronously, where the calling function awaits for a return value.
异步直接调用导致了基于编排的 FaaS 解决方案。一个函数只是根据其业务逻辑调用下一个函数,然后由该函数和 FaaS 框架来处理后续步骤,包括任何失败或错误。异步直接调用函数拓扑是将函数调用链接在一起的简单方法。图 9-4给出了一个示例。
Asychronous direct calls lead to a choreography-based FaaS solution. One function simply invokes the next one based on its business logic, leaving it up to that function and the FaaS framework to handle the next steps, including any failures or errors. An asynchronous direct-call function topology is a simple way to chain function calls together. Figure 9-4 illustrates an example.
函数 A 在处理一批事件时调用函数 B,完成后,函数 A 可以简单地更新其使用者偏移量并终止。同时,函数 B 继续其处理并向输出事件流产生任何输出。
Function A invokes function B as it processes its batch of events, and once done, function A can simply update its consumer offsets and terminate. Meanwhile, function B continues its processing and produces any outputs to the output event stream.
异步直接调用的一个主要缺点是确保仅在成功处理的情况下更新消费者偏移量。在示例中,函数 B 没有对函数 A 的反馈,因此只有函数 A 中的错误才能防止工作流错误地提交消费者组偏移量。但是,丢失事件对于某些工作流程可能并不重要,在这些情况下,可以 忽略此问题。
One major downside to asynchronous direct calls is in ensuring that the consumer offsets are updated only in the case of successful processing. In the example, function B has no feedback to function A, and so only errors in function A will prevent the workflow from incorrectly committing the consumer group offsets. However, losing events may not matter to some workflows, and in those cases this problem can be dismissed.
另一个潜在的主要问题是,由于多次调用函数 B,事件可能会被无序处理。考虑函数 A 的代码:
Another potentially major issue is that events may be processed out of order due to multiple invocations of function B. Consider the code for function A:
publicintfunctionA(Event[]events,Contextcontext){for(Eventevent:events){//Do function A's processing work//Invoke function B asynchronously per event//Does not wait for return valueasyncfunctionB(event);}context.success();return0;}
publicintfunctionA(Event[]events,Contextcontext){for(Eventevent:events){//Do function A's processing work//Invoke function B asynchronously per event//Does not wait for return valueasyncfunctionB(event);}context.success();return0;}
函数 B 是与函数 A 的工作内联调用的。根据您的 FaaS 框架,这可能会导致创建函数 B 的多个实例,每个实例都独立于其他实例运行。这将产生竞争条件,其中函数 B 的某些执行将先于其他执行完成,从而可能导致事件处理无序。
Function B is called inline with the work from function A. Depending on your FaaS framework, this may result in multiple instances of function B being created, each of which runs independently of the others. This will create a race condition where some executions of function B will finish before others, potentially leading to out-of-order processing of events.
同样,按如下方式编写代码也无法解决排序问题。处理仍然会无序发生,因为函数 A 的处理工作将在函数 B 执行之前对批处理中的所有事件执行。
Similarly, writing your code as follows will not solve ordering problems either. Processing will still happen out of order, as function A’s processing work will be executed for all events in the batch prior to function B’s execution.
publicintfunctionA(Event[]events,Contextcontext){for(Eventevent:events){//Do function A's processing work}//Invoke function B asynchronously with entire batch of eventsasyncFunctionB(events);context.success()return0;}
publicintfunctionA(Event[]events,Contextcontext){for(Eventevent:events){//Do function A's processing work}//Invoke function B asynchronously with entire batch of eventsasyncFunctionB(events);context.success()return0;}
对于每个事件,按顺序处理要求在处理下一个事件之前严格执行函数 A,然后执行 B。一个事件必须处理完毕才能开始下一个事件;否则,可能会导致不确定的行为。如果函数 A 和 B 作用于同一外部数据存储,则尤其如此,因为函数 B 可能依赖于函数 A 写入的数据。
In-order processing requires strictly executing function A before B, for each event, before processing the next event. An event must be completely processed before the next one can be started; otherwise, nondeterministic behavior will likely result. This is particularly true if functions A and B are acting on the same external data store, as function B may rely on data written by function A.
在许多情况下,异步调用不足以满足有界上下文的需求。在这些情况下,请考虑以编排方式使用同步调用是否更合适。
In many cases, asynchronous calls are not sufficient for the needs of the bounded context. In these cases, consider whether using synchronous calls in an orchestrated manner is more suitable.
同步函数调用允许您调用其他函数并等待结果,然后再继续其余的业务逻辑。这允许实现编排模式,如第 8 章所述。
Synchronous function calls allow you to invoke other functions and await the results before proceeding with the remaining business logic. This allows for the implementation of the orchestration pattern, as covered in Chapter 8.
在以下示例中,当新事件到达分区事件流时,将触发单个编排函数。该函数启动并开始处理输入的事件批次,为每个函数顺序调度事件。图 9-5显示了单个有界上下文中基于函数的编排示例。
In the following example, a single orchestration function is triggered when new events arrive in a partitioned event stream. The function starts up and begins processing the input batch of events, dispatching event sequentially for each function. Figure 9-5 shows an example of function-based orchestration within a single bounded context.
以下是编排代码的示例:
Here is an example of the orchestration code:
publicintorchestrationFunction(Event[]events,Contextcontext){for(Eventevent:events){//Synchronous function CallsResultresultFromA=invokeFunctionA(event);ResultresultFromB=invokeFunctionB(event,resultFromA);Outputoutput=composeOutputEvent(resultFromA,resultFromB);//Write to the output streamproducer.produce("Output Stream",output);}//This will tell the FaaS framework to update the consumer offsetscontext.success();return0;}
publicintorchestrationFunction(Event[]events,Contextcontext){for(Eventevent:events){//Synchronous function CallsResultresultFromA=invokeFunctionA(event);ResultresultFromB=invokeFunctionB(event,resultFromA);Outputoutput=composeOutputEvent(resultFromA,resultFromB);//Write to the output streamproducer.produce("Output Stream",output);}//This will tell the FaaS framework to update the consumer offsetscontext.success();return0;}
编排函数按顺序调用函数 A 和 B,并等待每个函数的结果。如果需要,可以将函数 A 的输出发送到 B。每个事件在下一个事件开始之前都通过工作流程进行完全处理,确保遵守偏移顺序。一旦消费者函数完成处理批次中的每个事件,它就可以发出成功消息并相应地更新偏移量。
The orchestration function invokes functions A and B in order and awaits the results from each function. The output of function A can be sent to B if needed. Each event is fully processed through the workflow before the next one is started, ensuring that offset order is respected. Once the consumer function has completed processing each event in the batch, it can issue a success message and update the offsets accordingly.
如果您使用具有单独提交功能的队列,则触发机制可以简单地为每个事件触发单独的编排功能。协调器完成工作后需要将处理确认提交回队列。如果编排器无法处理工作,它将由下一个创建的编排器实例来接替。
If you use a queue with individual commit capabilities, the triggering mechanism can simply trigger an individual orchestration function for each event. The orchestrator will need to commit the processing confirmation back to the queue after it has completed its work. In the case that the orchestrator fails to process the work, it will simply be picked up by the next orchestrator instance created.
一旦函数完成其工作或达到其分配的生命周期(通常在 5-10 分钟范围内)结束,该函数就会终止。函数实例被挂起并进入休眠状态,可以立即恢复。由于资源或时间限制,挂起的函数最终也可能从休眠缓存中逐出。
A function is terminated once it has completed its work or it reaches the end of its allocated lifespan, generally in the range of 5–10 minutes. The function instance is suspended and enters a state of hibernation, where it may be immediately revived. The suspended function may also eventually be evicted from the hibernation cache due to resource or time constraints.
您需要决定如何处理在函数终止之前对函数进行的任何打开的连接和资源分配。在消费者客户端的情况下,可以为函数实例分配特定的事件流分区。终止函数实例而不撤销这些分配可能会导致处理延迟,因为在达到超时之前不会重新分配使用者组的所有权。在执行消费者组重新平衡或终止的函数实例重新联机并恢复处理之前,不会处理来自这些分区的事件。
You will need to decide how to handle any open connections and resource assignments made to a function prior to its termination. In the case of a consumer client, the function instance may be assigned specific event stream partitions. Terminating the function instance without revoking these assignments may result in processing delays, as ownership in the consumer group won’t be reassigned until a timeout is reached. Events from those partitions won’t be processed until a consumer group rebalance is performed, or the terminated function instance comes back online and resumes processing.
如果您的函数几乎始终在线并处理事件,则可能不需要关闭连接和重新平衡消费者组。该函数可能只会在其生命周期结束时暂时挂起,短暂进入休眠状态,然后立即恢复到运行时。相反,对于仅间歇性运行的使用者函数,最好关闭所有连接并放弃事件流分区的分配。无论是热启动还是冷启动,下一个函数实例都必须重新创建连接。当有疑问时,清理连接通常是一个好主意;它减轻了外部数据存储和事件代理的负载,并减少了挂起的函数声明分区所有权的机会。
If your function is almost always online and processing events, closing connections and rebalancing the consumer group may not be necessary. The function will likely only be momentarily suspended at the end of it lifespan, go briefly into hibernation, and be restored to runtime immediately. Conversely, for a consumer function that runs only intermittently, it is best to close down all connections and relinquish assignment of the event stream partitions. The next function instance will have to re-create the connection regardless of whether it’s a warm start or cold start. When in doubt, cleaning up connections is generally a good idea; it lightens the load on external data stores and on the event broker and reduces the chances that suspended functions are laying claim to partition ownership.
每个职能根据其工作量都有特定的需求。优化函数在执行期间使用的资源可以确保保持较高的性能,同时保持较低的成本。在建立资源和调整函数参数时需要考虑一些事项。
Each function has specific needs based on its workload. Optimizing the resources that a function uses during its execution can ensure that performance remains high while costs remain low. There are a few things to consider when establishing the resources and tuning the parameters of your functions.
每个函数都可以分配特定数量的 CPU 和内存。根据您的功能需求调整这些参数非常重要;过度分配的成本可能很高,而分配不足可能会导致函数崩溃或需要很长时间才能完成。
Each function can be allocated a specific amount of CPU and memory. It is important to tune these parameters to the needs of your function; overallocation can be expensive, while underallocation may result in your functions crashing or taking too long to complete.
最大执行时间是另一个因素,因为它限制了函数可以运行的时间。该参数与批量大小密切相关,因为函数处理事件所需的时间通常与要处理的事件数量线性相关。将最大执行时间设置为高于处理特定批量大小的事件的最大预期时间,以避免不必要的函数超时错误。
Maximum execution time is another factor, as it limits how long a function may run. This parameter is closely related to the batch size, as the time a function needs to process events is very often linearly related, on average, with the number of events to process. Set the maximum execution time higher than the maximum expected time to process a specific batch size of events to avoid unnecessary function timeout errors.
最后,您必须考虑属于基于函数的解决方案的有界上下文中的状态存储的任何外部 I/O。函数的工作负载随输入事件流的变化而变化,某些工作负载需要与外部状态一致的 I/O,而其他工作负载仅需要零星的 I/O。无法提供足够的 I/O 资源可能会导致吞吐量和性能下降。
Lastly, you must consider any external I/O to state stores belonging within the bounded context of the function-based solution. The workload of a function varies with the flow of input events, with some workloads requiring consistent I/O to external state, and other workloads requiring only sporadic I/O. A failure to provide sufficient I/O resources can result in degraded throughput and performance.
如果函数在其执行生命周期内无法处理为其分配的一批事件,则该函数的执行将被视为失败,并且必须重新处理该批事件。但是,除非对函数分配的执行时间或输入事件的批量大小进行任何更改,否则它很可能会再次失败。因此,必须发生以下两种情况之一:
If a function is unable to process its assigned batch of events during its execution lifespan, the execution of the function is considered to have failed and the batch must be processed again. However, barring any changes to the function’s allocated execution time or the batch size of input events, it is likely to simply fail again. Therefore, one of two things must occur:
增加函数的最大执行时间。
Increase the maximum execution time of the function.
减小函数处理的事件的最大批量大小。
Decrease the maximum batch size of events processed by the function.
与事件代理建立自己的连接并管理事件消耗的函数也可以在执行期间定期提交偏移量,以确保批处理的部分完成。当函数传递一批消费事件时,这不起作用,因为它无法在处理期间更新消费者偏移量。
Functions that establish their own connections to the event broker and manage the consumption of events can also periodically commit offsets during execution, ensuring partial completion of the batch. This does not work when the function is passed the batch of consumed events, as it has no way to update the consumer offsets during processing.
此外,某些事件侦听器触发系统(例如 Amazon 和 Microsoft 提供的系统)允许您选择在失败时自动将批处理大小减半并重新执行失败的函数。随后的错误会导致输入批次再次减半并重新执行函数,直到达到可以按时完成处理的程度。
Additionally, some event-listener triggering systems, such as those provided by Amazon and Microsoft, give you the option to automatically halve the batch size on failure and re-execute the failed function. Subsequent errors result in the input batch being halved again and the function re-executed, until it reaches the point where it can complete its processing on time.
FaaS 解决方案为工作并行化提供了卓越的功能,特别是对于数据处理顺序并不重要的队列和事件流。对于分区事件流,如果事件的顺序确实很重要,则最大并行级别受到事件流中分区数量的限制,就像所有微服务实现一样。
FaaS solutions provide exceptional capabilities for the parallelization of work, especially for queues and event streams where the order in which data is processed is not important. For partitioned event streams, if the order of events is indeed important, the maximum level of parallelization is limited by the number of partitions in your event streams, just as it is for all microservice implementations.
扩展策略通常是 FaaS 框架的领域,因此请检查您的框架文档以了解提供了哪些选项。典型的选项包括根据消费者输入滞后、一天中的时间、处理吞吐量和性能特征进行扩展。
Scaling policies are typically the domain of the FaaS framework, so check your framework documentation to see what options are offered. Typical options involve scaling based on consumer input lag, time of day, processing throughput, and performance characteristics.
对于实例化和管理自己的事件代理连接的函数,请注意消费者进入或离开消费者组时分区分配重新平衡的影响。如果消费者频繁加入和离开消费者组,则消费者组最终可能会处于近乎不断的重新平衡状态,从而阻碍取得进展。在极端情况下,可能会陷入重新平衡的虚拟死锁,其中函数在其生命周期中反复分配和删除分区。当您使用许多具有较小消费者批量大小的短期函数时,可能会出现此问题,并且如果扩展对延迟过于敏感的策略,则可能会使问题变得更糟。
For functions that instantiate and manage their own event broker connections, beware the impact of partition assignment rebalancing when a consumer enters or leaves the consumer group. A consumer group can end up in a state of near-constant rebalancing if consumers are frequently joining and leaving the consumer group, preventing progress from being made. In extreme circumstances it is possible to get stuck in a virtual deadlock of rebalancing, where the functions spend their life cycle repeatedly having partitions assigned and removed. This problem can occur when you use many short-lived functions with small consumer batch sizes and can be made worse by scaling policies that are overly sensitive to delay. Instituting a step-based scaling policy or using a hysteresis loop can provide sufficient scaling responsiveness without putting the consumer group into a state of excessive rebalancing.
静态分区分配消除了动态分配的消费者组的重新平衡开销,并且还可以用于对事件流进行共同分区。函数首先会预先知道它们将从哪些分区中消费;不会有重新平衡,并且只要触发函数就可以简单地消耗事件。这种方法确实需要更仔细地考虑您的函数正在执行的工作,因为您需要确保每个分区都被使用。
Static partition assignments eliminate the rebalancing overhead of dynamically assigned consumer groups and can also be used to copartition event streams. Functions will start with the foreknowledge of which partitions they will consume from; there will be no rebalancing, and events can simply be consumed whenever the function is triggered. This approach does require a more careful consideration of the work that your function is performing, as you need to ensure that each partition is being consumed.
功能即服务是云计算的一个快速发展的领域。许多 FaaS 框架提供各种功能开发、管理、部署、触发、测试和可扩展性工具,允许您使用功能构建微服务。函数可以由事件流中的新事件、消费者组滞后状态、挂钟时间或自定义逻辑触发。
Function-as-a-Service is an area of cloud computing that is growing rapidly. Many FaaS frameworks offer a variety of function development, management, deployment, triggering, testing, and scalability tools that allow you to build your microservices using functions. Functions can be triggered by new events in an event stream, consumer group lag status, wall-clock time, or custom logic.
基于函数的解决方案在处理不需要事件调度的无状态和简单有状态业务问题时特别有用。编排模式允许按严格的顺序调用多个函数,同时也尊重事件流中的事件顺序。由于 FaaS 框架空间正在快速增长和发展,因此跟上您和您的组织感兴趣的平台的最新功能非常重要。
Function-based solutions are particularly useful in handling stateless and simple stateful business problems that do not require event scheduling. The orchestration pattern allows multiple functions to be called in strict sequential order, while also respecting the order of events from the event stream. Since the FaaS framework space is growing and evolving rapidly, it’s important to keep up with the newest features of the platforms of interest to you and your organization.
基本生产者和消费者 (BPC) 微服务从一个或多个事件流中摄取事件,应用任何必要的转换或业务逻辑,并将任何必要的事件发出到输出事件流。同步请求-响应 I/O 也可能是此工作流程的一部分,但该主题将在第 13 章中进行更详细的介绍。本章重点关注事件驱动组件。
Basic producer and consumer (BPC) microservices ingest events from one or more event streams, apply any necessary transformations or business logic, and emit any necessary events to output event streams. Synchronous request-response I/O may also be a part of this workflow, but that topic is covered in more detail in Chapter 13. This chapter focuses strictly on event-driven components.
BPC 微服务的特点是使用基本的消费者和生产者客户端。基本消费者客户端不包括任何事件调度、水印、物化机制、变更日志或具有本地状态存储的处理实例的水平扩展。这些功能通常只属于功能更齐全的框架,第 11 章和第 12章将更深入地讨论这些框架。虽然您当然可以开发自己的库来提供这些功能,但这超出了本章的范围。因此,您必须仔细考虑 BPC 模式是否适合您的业务需求。
BPC microservices are characterized by the use of basic consumer and producer clients. Basic consumer clients do not include any event scheduling, watermarks, mechanisms for materialization, changelogs, or horizontal scaling of processing instances with local state stores. These capabilities typically belong only to more full-featured frameworks, which Chapters 11 and 12 will discuss in more depth. While it is certainly possible for you to develop your own libraries to provide these features, doing this is quite beyond the scope of this chapter. Thus, you must carefully consider if the BPC pattern will work for your business requirements.
生产者和消费者客户端可以使用最常用的语言轻松获得,从而降低了开始使用事件驱动的微服务时的认知开销。有界上下文的整个工作流程包含在单个微服务应用程序的代码中,使微服务的职责保持本地化且易于理解。工作流还可以轻松地包装到一个或多个容器中(取决于实现的复杂性),然后可以使用微服务的容器管理解决方案进行部署和 执行。
Producer and consumer clients are readily available in most commonly used languages, lowering the cognitive overhead in getting started with event-driven microservices. The entire workflow of the bounded context is contained within the code of the single microservice application, keeping the responsibilities of the microservice localized and easy to understand. The workflow can also easily be wrapped into one or more containers (depending on the complexity of the implementation), which can then be deployed and executed with the microservice’s container management solution.
尽管缺乏大部分功能齐全的框架组件,BPC 微服务仍可以满足广泛的业务需求。无状态转换等简单模式很容易实现,不需要确定性事件调度的有状态模式也是如此。
BPC microservices can fulfill a wide range of business requirements despite lacking most of the full-featured framework components. Simple patterns such as stateless transformations are easily implemented, as are stateful patterns where deterministic event scheduling is not required.
在 BPC 实现中,外部状态存储比内部状态存储更常用,因为如果没有功能齐全的流框架,在多个实例之间扩展本地状态并从实例故障中恢复是很困难的。外部状态存储可以为多个微服务实例提供统一的访问以及数据备份和恢复机制。
External state stores are more commonly used than internal state stores in BPC implementations, as scaling local state between multiple instances and recovering from instance failures is difficult without a full-featured streaming framework. External state stores can provide multiple microservice instances with uniform access as well as data backup and recovery mechanisms.
让我们看一些基本 BPC 实现效果特别好的用例。
Let’s look at a few use cases in which basic BPC implementations work particularly well.
遗留系统可以通过将基本的生产者/消费者客户端集成到其代码库中来参与事件驱动的架构。这种集成通常在采用事件驱动的微服务的早期就开始,甚至可能是将遗留系统引导到事件驱动的生态系统的策略的一部分(请参阅第 4 章)。遗留系统根据需要向事件代理的单一事实源生成自己的数据,并从其他事件流消耗回所需的任何事件。
Legacy systems can participate in event-driven architectures by integrating a basic producer/consumer client into their codebase. This integration often begins early in the adoption of event-driven microservices and may even be part of your strategy for bootstrapping legacy systems into the event-driven ecosystem (see Chapter 4). Legacy systems produce their own data to the event broker’s single source of truth as required and consume back any events that they need from other event streams.
在某些情况下,无法安全地修改旧代码库以生成和使用事件流中的数据。sidecar模式特别适用于这种场景,因为它支持一些事件驱动的功能而不影响源代码库。
In some scenarios it’s not possible to safely modify the legacy codebase to produce and consume data from event streams. The sidecar pattern is particularly applicable to this scenario, as it enables some event-driven functionality without affecting the source codebase.
电子商务商店有一个显示其包含的所有库存和产品数据的前端。以前,前端服务将通过使用计划的批处理作业与只读从属数据存储同步来获取其所有数据,如图10-1所示。
An ecommerce store has a frontend that displays all the stock and product data it contains. Previously, the frontend service would source all of its data by synchronizing with a read-only subordinate data store using a scheduled batch job, as in Figure 10-1.
目前,有两个事件流,一个包含产品信息,另一个包含产品库存水平。您可以使用 sidecar 实现将此数据放入数据存储中,其中 BPC 消费事件并将它们插入到关联的数据集中。前端可以访问产品更新的近实时数据源,而无需更改任何系统代码,如图10-2所示。
Today, there are two event streams, one with product information and one with product stock levels. You can use a sidecar implementation to sink this data into the data store, where a BPC consumes the events and upserts them into the associated data sets. The frontend gains access to a near-real time data feed of product updates, without having to change any of the system code, as in Figure 10-2.
sidecar 驻留在其自己的容器内,但必须是前端服务的单个可部署组件的一部分。必须执行额外的测试以确保集成边车按预期运行。sidecar 模式允许您向系统添加新功能,而无需对遗留代码库进行重大更改。
The sidecar resides inside its own container but must be part of the single deployable of the frontend service. Additional tests must be performed to ensure that the integrated sidecar operates as expected. The sidecar pattern allows you to add new functionality to a system without requiring significant changes to the legacy codebase.
许多业务流程对事件到达的顺序没有任何要求,但确实要求所有必要的事件最终到达。这称为门控模式,BPC 在其中可以很好地实现。
Many business processes do not have any requirements regarding the order in which events arrive, but do require that all necessary events eventually arrive. This is known as a gating pattern and is one in which the BPC works well as an implementation.
假设您为图书出版商工作,在书籍准备好发送到印刷厂之前必须完成三件事。这些事件发生的顺序并不重要,但重要的是每个事件发生在将书发布到印刷商之前:
Say that you work for a book publisher and there are three things that must be done before a book is ready to be sent to the printer. It is not important in which order these events occur, but it is important that each one occurs prior to releasing the book to the printers:
书上的内容一定是写过的。
The contents of the book must have been written.
这本书的封面艺术一定已经创作好了。
The cover art for the book must have been created.
价格必须根据地区和格式设定。
The prices must be set according to regions and formats.
每个事件流都充当逻辑驱动程序。当新事件出现在这些流中的任何一个上时,它首先在其适当的表中具体化,然后用于查找每个其他表以查看其他事件是否存在。图 10-3给出了一个示例。
Each of these event streams acts as a driver of logic. When a new event comes in on any of these streams, it is first materialized in its proper table and subsequently used to look up every other table to see if the other events are present. Figure 10-3 illustrates an example.
在此示例中,以 ISBN 0010 结尾的书籍将已发布到输出书籍事件流。同时,以 ISBN 0011 结尾的书目前正在等待封面艺术可用,尚未发布到输出流。
In this example, the book ending with ISBN 0010 will already have been published to the output book event stream. Meanwhile, the book ending with ISBN 0011 is currently waiting for cover art to be available and has not been published to the output stream.
门控模式中也可能需要人类的明确批准。“示例:报纸出版工作流程(审批模式)”对此进行了更详细的介绍。
Explicit approval from a human being may also be required in the gating pattern. This is covered in more detail in “Example: Newspaper publishing workflow (approval pattern)”.
当底层数据层执行大部分业务逻辑(例如地理空间数据存储)时,BPC 也是一种合适的方法;自由文本搜索;以及机器学习、人工智能和神经网络应用。电子商务公司可以提取从网站上抓取的新产品,并使用 BPC 微服务执行分类,后端数据层是经过批量训练的机器学习分类器。或者,用户行为事件(例如打开应用程序)可以与地理空间数据存储相关联,以确定显示广告的最近的零售商。在这些场景中,处理事件的复杂性完全转移到底层数据层,生产者和消费者组件充当简单的集成机制。
The BPC is also a suitable approach when the underlying data layer performs most of the business logic, such as a geospatial data store; free-text search; and machine learning, AI, and neural network applications. An ecommerce company may ingest new products scraped from websites and perform classification using a BPC microservice, with the backend data layer being a batch-trained machine learning categorizer. Alternately, user behavior events, such as opening the application, may be correlated with a geospatial data store to determine the nearest retailers from which to show advertisements. In these scenarios, the complexity of processing the event is offloaded entirely to the underlying data layer, with the producer and consumer components acting as simple integration mechanisms.
微服务的处理需求和数据存储需求并不总是线性相关的。例如,微服务必须处理的事件量可能随时间而变化。以下示例中包含的一种常见负载模式反映了当地人群的睡眠/觉醒周期,白天活动密集,夜间活动非常低。
The processing needs and the data storage needs of a microservice are not always linearly related. For instance, the volume of events that a microservice must process may vary with time. One common load pattern, which is incorporated into the following example, mirrors the sleep/wake cycle of a local population, with intensive activity during the day and very low activity during the night.
考虑这样一个场景:用户行为事件聚合到 24 小时会话中。这些会话中的数据用于确定哪些产品最近最受欢迎,进而用于推动销售广告。一旦 24 小时聚合会话完成,它就会从数据存储中刷新并发送到输出事件流,从而释放数据存储空间。每个用户都有一个在外部数据存储中维护的聚合,如下所示:
Consider a scenario where user behavior events are aggregated into 24-hour sessions. The data from these sessions is used to determine which products are the most recently popular and in turn used to drive advertisements for sales. Once the 24-hour aggregation session is completed, it is flushed from the data store and emitted to an output event stream, freeing up the data storage space. Each user has an aggregation maintained in an external data store, which looks something like this:
| 钥匙 | 价值 |
|---|---|
|
|
服务的处理需求随着使用该产品的人的睡眠/觉醒周期而变化。在夜间,当大多数用户都在睡觉时,与白天所需的处理能力相比,执行聚合所需的处理能力非常少。为了节省处理能力的费用,服务在夜间缩减了规模。
The processing needs of the service change with the sleep/wake cycles of the people using the product. At nighttime, when most users are asleep, very little processing power is needed to perform the aggregations compared to what’s required during the day. In the interest of saving money on processing power, the service is scaled down in the night.
分区分配器可以将输入事件流分区重新分配给单个处理器实例,因为它可以处理所有用户事件的消耗和处理。请注意,尽管事件量较低,但潜在用户的域保持不变,因此该服务需要对所有用户聚合的完全访问权限。缩小处理规模不会影响服务必须维护的状态大小。
The partition assignor can reassign the input event stream partitions to a single processor instance, as it can handle the consumption and processing of all user events. Note that despite the volume of events being low, the domain of potential users remains constant and so the service requires full access to all user aggregations. Scaling down the processing has no impact on the size of state that the service must maintain.
白天,可以使额外的处理实例联机以处理增加的事件负载。在这种特定场景中,数据存储的查询速率也会增加,但缓存、分区和批处理可以帮助保持负载比处理要求的线性增加更轻。
During the day, additional processing instances can be brought online to handle the increased event load. The query rate of the data store will also increase in this particular scenario, but caching, partitioning, and batching can help keep the load lighter than the linear increase in processing requirements.
BPC 微服务还可以利用外部流处理系统来完成原本很难在本地完成的工作。这是一种混合应用程序模式,业务逻辑分布在 BPC 和外部流处理框架之间。第 11 章的重量级框架是很好的候选者,因为它们可以通过简单的集成提供大规模流处理。
BPC microservices can also leverage external stream-processing systems to do work that may otherwise be too difficult to do locally. This is a hybrid application pattern, with business logic spread between the BPC and the external stream-processing framework. The heavyweight frameworks of Chapter 11 are excellent candidates for this, as they can provide large-scale stream processing with simple integrations.
BPC 实现可以执行原本无法执行的操作,同时仍然可以访问任何必要的语言功能和库。例如,您可以使用外部流处理框架跨多个事件流执行复杂的聚合,同时使用 BPC 微服务将结果填充到本地数据存储并提供请求-响应查询。
The BPC implementation can perform operations that would otherwise be unavailable to it, while still having access to any necessary language features and libraries. For example, you could use the external stream-processing framework to perform complex aggregations across multiple event streams, while using the BPC microservice to populate a local data store with the results and serve up request-response queries.
假设您的 BPC 服务需要利用流处理框架的联接功能,该框架特别擅长联接大量物化事件流。外部流处理器将简单地将事件流具体化到表中,并将具有相同键的那些行连接在一起。这个简单的连接操作如图 10-4所示。
Say your BPC service needs to leverage the joining capabilities of a stream-processing framework, which is particularly good at joining large sets of materialized event streams. The external stream processor will simply materialize event streams into tables and join those rows together that have the same key. This simple join operation is shown in Figure 10-4.
混合BPC需要使用兼容的客户端来启动外部流处理框架的工作。该客户端会将代码转换为框架的指令,框架本身将处理事件的消费、连接和生成到连接的输出事件流中。此设计将工作外包给外部处理服务,该服务将以事件流的形式返回结果。其工作流程如图10-5所示。
The hybrid BPC needs to use a compatible client to start up the work on the external stream processing framework. This client will transform the code into instructions for the framework, which will itself handle consuming, joining, and producing events into the joined output event stream. This design outsources the work to an external processing service that will return the results in the form of an event stream. The workflow for this would look like Figure 10-5.
BPC实例化一个客户端来运行外部流处理工作。当 BPC 终止时,外部流处理实例也应终止,以确保没有幽灵进程仍在运行。
The BPC instantiates a client to run the external stream processing work. When the BPC is terminated, the external stream processing instance should also be terminated to ensure that no ghost processes are left running.
此模式的主要优点是它解锁了微服务可能无法使用的流处理功能。可用性仅限于具有相应流处理客户端的语言,并且并非所有语言都支持所有功能。这种模式经常与轻量级和重量级框架一起使用;例如,它是基于 SQL 的流操作的主要用例之一,例如Confluence 的 KSQL提供的操作。诸如此类的技术选项提供了一种增强BPC 产品的方法,使其能够获得原本无法获得的强大流处理选项。
The major advantage of this pattern is that it unlocks stream-processing features that may otherwise be unavailable to your microservice. Availability is limited to languages with corresponding stream-processing clients, and not all features may be supported for all languages. This pattern is frequently used in tandem with both lightweight and heavyweight frameworks; for example, it’s one of the primary use cases for SQL-based stream operations, such as those provided by Confluent’s KSQL. Technology options like these provide a way to augment the offerings of the BPC, giving it access to powerful stream-processing options that it would not have otherwise.
这种模式的主要缺点与复杂性的增加有关。测试应用程序变得更加复杂,因为您还必须找到一种将外部流处理框架集成到测试环境中的方法(请参阅第 15 章)。调试和开发的复杂性也会增加,因为流框架的引入增加了移动部件的数量和潜在的错误。最后,处理微服务的有界上下文可能会变得更加困难,因为您需要确保可以通过混合应用程序部署轻松管理部署、回滚和操作。
The main drawbacks of this pattern relate to the increase in complexity. Testing the application becomes much more complex, as you must also find a way to integrate the external stream-processing framework into your testing environment (see Chapter 15). Debugging and development complexity also increase, because the introduction of the streaming framework increases the number of moving parts and potential for bugs. Finally, handling the bounded context of the microservice may become more difficult, as you need to ensure that you can easily manage the deployment, rollback, and operations with hybrid application deployments.
BPC 模式简单但功能强大。它构成了许多无状态和有状态事件驱动的微服务模式的基础。您可以使用 BPC 模式轻松实现无状态流和简单的有状态应用程序。
The BPC pattern is simple yet powerful. It forms the foundation of many stateless and stateful event-driven microservice patterns. You can easily implement stateless streaming and simple stateful applications using the BPC pattern.
BPC 模式也很灵活。它与数据存储层完成大部分业务工作的实现完美匹配。您可以将其用作事件流和遗留系统之间的接口层,并利用外部流处理系统来增强其功能。
The BPC pattern is also flexible. It pairs well with implementations where the data storage layer does most of the business work. You can use it as an interfacing layer between event streams and legacy systems, as well as leverage external stream processing systems to augment its capabilities.
然而,由于其基本性质,它确实需要您投资库来访问简单状态实现、事件调度和基于时间戳的决策等机制。这些组件与第 11 章和第 12章中的产品交叉,因此您需要决定要进行多少内部开发,而不是采用更多专用解决方案。
Due to its basic nature, however, it does require you to invest in libraries to access mechanisms such as simple state materialization, event scheduling, and timestamp-based decision making. These components intersect with the offerings found in Chapters 11 and 12, and thus you will need to decide how much in-house development you would like to do versus adopting more purpose-built solutions.
本章和下一章将介绍事件驱动处理中最常用的全功能框架。它们通常称为流框架,提供用于处理数据流的机制和 API,并且通常用于向事件代理消费和生成事件。这些框架大致可以分为本章介绍的重量级框架和下一章介绍的轻量级框架。这些章节并不是为了比较这些技术,而是为了提供这些框架如何工作的一般概述。然而,某些部分研究了特定于框架的功能,特别是当它们与以类似微服务的方式实现应用程序有关时。为了评估重量级框架,本章涵盖了Apache Spark、Apache Flink、Apache Storm、Apache Heron和Apache Beam 模型的各个方面,作为常见提供的技术和操作类型的示例。
This chapter and the next cover the full-featured frameworks most commonly used in event-driven processing. Frequently referred to as streaming frameworks, they provide mechanisms and APIs for handling streams of data and are often used for consuming and producing events to an event broker. These frameworks can be roughly divided into heavyweight frameworks, which are covered in this chapter, and lightweight frameworks, covered in the next. These chapters aren’t meant to compare the technologies, but rather to provide a generalized overview of how these frameworks work. However, some sections examine framework-specific features, especially as they pertain to implementing applications in a microservice-like way. For the purposes of evaluating heavyweight frameworks, this chapter covers aspects of Apache Spark, Apache Flink, Apache Storm, Apache Heron, and the Apache Beam model as examples of the sorts of technology and operations commonly provided.
重量级流框架的一个定义特征是它需要独立的处理资源集群来执行其操作。该集群通常由许多可共享的工作节点以及一些调度和协调工作的主节点组成。此外,领先的 Apache 解决方案传统上依赖 Apache Zookeeper(另一个集群服务)来提供高可用性支持并协调集群领导者选举。尽管 Zookeeper 对于将重量级集群投入生产来说并不是绝对必要的,但如果您创建自己的集群,您应该仔细评估是否需要它。
One defining characteristic of a heavyweight streaming framework is that it requires an independent cluster of processing resources to perform its operations. This cluster typically constitutes a number of shareable worker nodes, along with some master nodes that schedule and coordinate work. Additionally, the leading Apache solutions traditionally rely on Apache Zookeeper, another clustered service, to provide high-availability support and coordinate cluster leader elections. Though Zookeeper is not absolutely essential for bringing a heavyweight cluster to production, you should carefully evaluate whether you need it should you create your own cluster.
第二个定义特征是重量级框架使用自己的内部机制来处理故障、恢复、资源分配、任务分配、数据存储、通信以及处理实例和任务之间的协调。这与轻量级框架、FaaS 和 BPC 实现形成鲜明对比, 这些实现严重依赖容器管理系统 (CMS) 和事件代理来实现这些功能。
A second defining characteristic is that the heavyweight framework uses its own internal mechanisms for handling failures, recovery, resource allocation, task distribution, data storage, communication, and coordination between processing instances and tasks. This is in contrast to the lightweight framework, FaaS, and BPC implementations that rely heavily on the container management system (CMS) and the event broker for these functions.
这两个特点是这些框架被称为“重量级”的主要原因。必须独立于事件代理和 CMS 来管理和维护额外的集群框架并不是一项小任务。
These two characteristics are the main reasons why these frameworks are dubbed “heavyweight.” Having to manage and maintain additional clustered frameworks independently of the event broker and the CMS is no small task.
一些重量级框架正在转向类似轻量级的执行模式。这些轻量级模式与用于操作其他微服务实现的 CMS 很好地集成。
Some heavyweight frameworks are moving toward lightweight-like execution modes. These lightweight modes integrate well with the CMS used to operate other microservice implementations.
您可能已经注意到,重量级框架执行的许多操作已经由 CMS 和事件代理处理。CMS 可以管理系统的资源分配、故障、恢复和扩展,而事件代理可以在单个微服务的实例之间提供基于事件的通信。重量级框架是合并了 CMS 和事件代理的功能的单一解决方案。我们将在下一章轻量级框架中进一步探讨这个主题。
You may have noticed that the heavyweight framework does a lot of things that are already handled by the CMS and event broker. The CMS can manage resource allocation, failures, recovery, and scaling of systems, while the event broker can provide event-based communication between the instances of a single microservice. The heavyweight framework is a single solution that merges those capabilities of the CMS and event broker. We’ll explore this topic a bit further in the next chapter on lightweight frameworks.
重量级流处理框架直接源自其重量级批处理前身。其中最广为人知的 Apache Hadoop 于 2006 年发布,为任何人提供开源大数据技术。Hadoop 将多种技术捆绑在一起,提供大规模并行处理、故障恢复、数据持久性和节点间通信,使用户能够廉价且轻松地访问商用硬件,以解决需要数千个节点(或更多)的问题。
Heavyweight stream-processing frameworks are directly descended from their heavyweight batch-processing predecessors. One of the most widely known, Apache Hadoop, was released in 2006, providing open source big-data technologies for anyone to use. Hadoop bundled a number of technologies together to offer massive parallel processing, failure recovery, data durability, and internode communication, allowing users to access commodity hardware cheaply and easily to solve problems requiring many thousands of nodes (or more).
MapReduce 是处理超大批量数据(又称大数据)的第一个广泛使用的方法之一,但是,虽然功能强大,但与当今的许多选项相比,它执行速度很慢。大数据的规模随着时间的推移稳步增长;尽管早期数百(或数千)GB 的工作负载很常见,但如今的工作负载已扩展到 TB 和 PB 范围。随着这些数据集的增长,对更快的处理、更强大的选项、更简单的执行选项以及可以提供近实时流处理功能的解决方案的需求也在增长。
MapReduce was one of the first widely available means of processing extremely large batches of data (aka big data), but, while powerful, it executes slowly in comparison to many of today’s options. The size of big data has steadily increased over time; although workloads of hundreds (or thousands) of gigabytes were common in the early days, workloads today have scaled to sizes in the terabyte and petabyte range. As these data sets have grown so has the demand for faster processing, more powerful options, simpler execution options, and solutions that can provide near-real-time stream-processing capabilities.
这就是 Spark、Flink、Storm、Heron 和 Beam 的用武之地。这些解决方案的开发目的是处理数据流,并比基于批处理的 MapReduce 作业更快地提供可操作的结果。其中一些(例如 Storm 和 Heron)是纯流技术,目前不提供批处理。Spark 和 Flink 等其他解决方案将批处理和流处理合并到一个解决方案中。
This is where Spark, Flink, Storm, Heron, and Beam come in. These solutions were developed to process streams of data and provide actionable results much sooner than those provided by batch-based MapReduce jobs. Some of these, like Storm and Heron, are streaming-only technologies and do not currently provide batch processing. Others, like Spark and Flink, merge batch and streaming processing into a single solution.
这些技术无疑为大多数大数据爱好者所熟悉,并且可能已经在许多组织的数据科学和分析部门中得到一定程度的使用。事实上,这就是许多组织开始涉足事件驱动处理的方式,因为这些团队将现有的基于批处理的作业转换为基于流的管道。
These technologies are undoubtedly familiar to most big-data aficionados and are likely already being used to some extent in the data science and analytics branches of many organizations. In fact, this is how many organizations start dabbling in event-driven processing, as these teams convert their existing batch-based jobs into streaming-based pipelines.
上述重量级开源 Apache 框架的运行方式都非常相似。专有解决方案,例如 Google 的 Dataflow,它执行使用 Apache Beam 的 API 编写的应用程序,可能以类似的方式运行,但这只是一个假设,因为源已关闭且后端未详细描述。详细描述重量级框架的挑战之一是每个框架都有自己的操作和设计细微差别,并且对每个框架的全面覆盖远远超出了本章的范围。
The aforementioned heavyweight open source Apache frameworks all operate in a fairly similar manner. Proprietary solutions, like Google’s Dataflow, which executes applications written using Apache Beam’s API, probably operate in a similar fashion, but this is only an assumption, given that the source is closed and the backend is not described in detail. One of the challenges of describing heavyweight frameworks in detail is that each has its own operational and design nuances, and full coverage of each framework is far beyond the scope of this chapter.
确保您彻底阅读并理解详细说明您的特定重量级框架如何 运行的文档。
Make sure that you thoroughly read and understand the documents detailing how your specific heavyweight framework operates.
重量级流处理集群是一组专用处理和存储资源,分为两个主要角色。第一个角色是主节点,它对执行者和工作人员执行的任务进行优先级排序、分配和管理。第二个角色是执行者,使用该工作节点可用的处理能力、内存、本地和远程磁盘来完成这些任务。在事件驱动的处理中,这些任务将连接到事件代理并使用事件流中的事件。图 11-1显示了其工作原理的粗略分解。
A heavyweight stream processing cluster is a grouping of dedicated processing and storage resources, broken down into two primary roles. The first role is the master node, which prioritizes, assigns, and manages executors and tasks performed by the workers. The second role, the executor, completes these tasks using the processing power, memory, local, and remote disk available to that worker node. In event-driven processing, these tasks will connect to the event broker and consume events from event streams. Figure 11-1 shows a rough breakdown of how this works.
该图还显示了 Apache Zookeeper,它为这个流集群起到了支撑作用。Zookeeper 提供高度可靠的分布式协调,用于确定哪个主节点负责(因为节点失败的情况并不少见,无论是工作节点、主节点还是 Zookeeper 节点)。当主节点发生故障时,Zookeeper 帮助确定剩余主节点中的哪一个是新的领导者,以确保操作的连续性。
This figure also shows Apache Zookeeper, which plays a supporting role for this streaming cluster. Zookeeper provides highly reliable distributed coordination and is used to determine which master node is in charge (as it is not uncommon for nodes to fail, be they workers, masters, or Zookeeper nodes). Upon failure of a master node, Zookeeper helps decide which of the remaining masters is the new leader to ensure continuity of operations.
Zookeeper 历来是提供分布式重量级框架协调的主要组件。较新的框架可能会也可能不会使用 Zookeeper。无论哪种情况,分布式协调对于可靠运行分布式工作负载都至关重要。
Zookeeper has historically been a major component in providing coordination of distributed heavyweight frameworks. Newer frameworks may or may not use Zookeeper. In either case, distributed coordination is essential for reliably running distributed workloads.
作业是使用框架的软件开发工具包 (SDK) 构建的流处理拓扑,旨在解决特定有界上下文的问题。它无限期地在集群上运行,在事件到达时对其进行处理,就像本书中描述的任何其他微服务一样。
A job is a stream-processing topology that is built using the framework’s software development kit (SDK) and designed to solve problems of the particular bounded context. It runs on the cluster indefinitely, processing events as they arrive, just like any other microservice described in this book.
集群接受后,定义的流处理拓扑将被分解为任务并分配给工作节点。任务管理器监视任务并确保它们完成。当发生故障时,任务管理器会在可用执行器之一中重新启动工作。任务管理器通常设置为高可用性,这样,如果任务管理器运行的节点发生故障,备份可以接管,从而防止所有正在运行的作业失败。
Upon acceptance by the cluster, the defined stream processing topology is broken down into tasks and assigned to the worker nodes. The task manager monitors the tasks and ensures that they are completed. When a failure occurs, the task manager restarts the work in one of the available executors. Task managers are usually set up with high-availability, such that if the node on which the task manager is operating fails, a backup can take over, preventing all running jobs from failing.
图 11-2显示了一个作业通过主节点 1 提交到集群,该作业又被转换为任务以供执行器处理。这些长时间运行的任务建立与事件代理的连接并开始使用事件流中的事件。
Figure 11-2 shows a job being submitted to the cluster via master node 1, which in turn is translated into tasks for processing by the executors. These long-running tasks establish connections to the event broker and begin to consume events from the event stream.
尽管此示例显示任务和流分区之间的 1:1 映射,但您可以配置您希望应用程序使用的并行度。一个任务可以从所有分区消耗数据,或者许多任务可以从同一个分区消耗数据,比如在队列的情况下。
Though this example shows a 1:1 mapping between tasks and stream partitions, you can configure the amount of parallelism that you would like your application to use. One tasks can consume from all the partitions, or many tasks could consume from the same partition, say in the case of a queue.
本章讨论的重量级框架主要是分析技术。它们在近乎实时地分析大量事件以实现更快的决策方面提供了重要价值。一些相当常见的使用模式包括:
The heavyweight frameworks discussed in this chapter are predominantly analytical technologies. They provide significant value around analyzing large volumes of events in near–real time to enable quicker decision making. Some fairly common patterns of usage include the following:
提取数据、转换数据并将其加载到新的数据存储 (ETL)
Extract data, transform it, and load it into a new data store (ETL)
执行基于会话和基于窗口的分析
Perform session- and window-based analysis
检测异常行为模式
Detect abnormal patterns of behavior
聚合流并维护状态
Aggregate streams and maintain state
执行任何类型的无状态流操作
Perform any sort of stateless streaming operations
这些框架功能强大且相当成熟,许多组织都在使用它们并为其源代码做出贡献。有大量书籍和博客文章、优秀的文档以及许多示例应用程序可供您使用。
These frameworks are powerful and fairly mature, with many organizations using them and contributing back to their source code. There are numerous books and blog posts, excellent documentation, and many sample applications available to you.
然而,有几个相当显着的缺点限制但不完全排除基于这些框架的微服务应用程序。
There are, however, several fairly significant shortcomings that limit, but not completely preclude, microservice applications based on these frameworks.
首先,这些重量级框架最初设计时并没有考虑微服务式部署。部署这些应用程序需要事件代理和 CMS 之外的专用资源集群,这增加了大规模管理大量应用程序的复杂性。有多种方法可以减轻这种复杂性,以及部署新技术的发展,本章稍后将详细介绍其中一些方法。
First, these heavyweight frameworks were not originally designed with microservice-style deployment in mind. Deploying these applications requires a dedicated resource cluster beyond that of the event broker and CMS, adding to the complexity of managing large numbers of applications at scale. There are ways to mitigate this complexity, along with new technological developments for deployment, some of which are covered in detail later in this chapter.
其次,这些框架大多数都是基于 JVM(Java 虚拟机)的,这限制了可用于创建单一微服务应用程序的实现语言。常见的解决方法是使用重量级框架作为其自己的独立应用程序来执行转换,而另一种语言的另一个独立应用程序则提供转换后的状态存储中的业务功能。
Second, most of these frameworks are JVM (Java Virtual Machine)-based, which limits the implementation languages you can use to create singular microservice applications. A common workaround is to use the heavyweight framework to perform transformations as its own standalone application, while another standalone application in another language serves the business functionality from the transformed state store.
第三,并非所有框架都支持将实体流具体化为无限期保留的表。这可能会妨碍创建表-表连接和流-表连接以及实现图 10-3中所示的门控模式等模式。
Third, materializing an entity stream into an indefinitely retained table is not supported out of the box by all frameworks. This can preclude creating table-table joins and stream-table joins and implementing patterns such as the gating pattern shown earlier in Figure 10-3.
即使重量级框架确实支持流具体化和连接,这在文档中通常也不会立即显现出来。其中许多框架主要关注基于时间的聚合,示例、博客文章和广告强调基于有限窗口大小的时间序列分析和聚合。一些仔细的挖掘表明,领先的框架提供了一个全局窗口,它允许事件流的具体化。从这里,您可以实现自己的自定义联接功能,但考虑到它们在组织中大规模处理事件流的重要性,我发现这些功能的记录和展示仍然远远不够充分。
Even when heavyweight frameworks do support stream materialization and joins, that is often not immediately apparent in the documentation. A number of these frameworks focus heavily on time-based aggregations, with examples, blog posts, and advertisements emphasizing time-series analysis and aggregations based on limited window sizes. Some careful digging reveals that the leading frameworks provide a global window, which allows for the materialization of event streams. From here, you can implement your own custom join features, though I find that these are still far less well documented and exhibited than they should be, considering their importance in handling event streams at scale in an organization.
同样,这些缺点表明了这些框架在设计和实施时所设想的分析工作负载类型。对各个实现的技术改进以及对独立于实现的通用 API(例如 Apache Beam)的投资正在推动重量级框架领域的不断变化,值得关注领导者以了解新版本带来的内容。
Again, these shortcomings are indicative of the types of analytical workloads that were envisioned for these frameworks when they were being designed and implemented. Technological improvements to individual implementations and investment into common APIs that are independent of implementation (e.g., Apache Beam) are driving continual changes in the heavyweight framework domain, and it is worth keeping an eye on the leaders to see what new releases bring.
在构建和管理重量级流处理集群时,有多种选择,每种选择都有自己的优点和缺点。
There are a number of options when it comes to building and managing your heavyweight stream processing cluster, each with its own benefits and drawbacks.
管理集群的第一种也是最简单的方法就是花钱请人为你做这件事。正如有许多计算服务提供商一样,也有一些提供商很乐意为您托管并可能管理您的大部分运营需求。与启动自己的集群的预计成本相比,这通常是花费的美元最昂贵的选项,但它显着降低了运营开销并消除了对内部专业知识的需求。例如,Amazon 同时提供托管的 Flink 和 Spark 服务;Google、Databricks和 Microsoft 提供自己的 Spark 捆绑包;Google 提供了 Dataflow,这是它自己的 Apache Beam 运行程序的实现。
The first, and simplest, way to manage a cluster is to just pay someone to do it for you. Just as there are a number of compute service providers, there are also providers who will be happy to host and possibly manage most of your operational needs for you. This is usually the most expensive option in terms of dollars spent when compared to the projected costs of starting your own cluster, but it significantly reduces the operational overhead and removes the need for in-house expertise. For example, Amazon offers both managed Flink and Spark services; Google, Databricks, and Microsoft offer their own bundling of Spark; and Google offers Dataflow, its own implementation of an Apache Beam runner.
关于这些服务需要注意的一件事是,它们似乎正在不断转向完全无服务器风格的方法,在这种方法中,作为订阅者,整个物理集群对您来说是不可见的。这可能会也可能不会被接受,具体取决于您的安全性、性能和数据隔离需求。请确保您了解这些服务提供商提供的内容和不提供的内容,因为它们可能不包含独立运营的集群的所有功能。
One thing to note about these services is that they seem to be continually moving toward a full serverless-style approach, where the entire physical cluster is invisible to you as a subscriber. This may or may not be acceptable depending on your security, performance, and data isolation needs. Be sure that you understand what is and is not offered by these service providers, as they may not include all of the features of an independently operated cluster.
重量级框架可能拥有自己独立于 CMS 的专用可扩展资源集群。这种部署是重量级集群的历史规范,因为它紧密模拟了原始 Hadoop 发行版。在需要大量(数百或数千)工作节点的服务使用重量级框架的情况下,这种情况很常见。
A heavyweight framework may have its own dedicated scalable resource cluster independent of the CMS. This deployment is the historical norm for heavyweight clusters, as it closely models the original Hadoop distributions. It is common in situations where the heavyweight framework will be used by services requiring a large number (hundreds or thousands) of worker nodes.
集群也可以与容器管理系统结合创建。第一种模式仅涉及在 CMS 提供的资源上部署集群,而第二种模式则涉及利用 CMS 本身作为扩展和部署单个作业的手段。在 CMS 上部署集群的一些主要好处是您可以获得它提供的监控、日志记录和资源管理功能。然后,扩展集群就变成了简单地添加或删除必要的节点类型的问题。
A cluster can also be created in conjunction with the container management system. The first mode involves just deploying the cluster on CMS-provisioned resources, whereas the second mode involves leveraging the CMS itself as the means of scaling and deploying individual jobs. Some of the major benefits of deploying your cluster on the CMS is that you gain the monitoring, logging, and resource management it provides. Scaling the cluster then becomes a matter of simply adding or removing the necessary node types.
使用 CMS 部署重量级集群有很多好处。主节点、工作节点和 Zookeeper(如果适用)在它们自己的容器或虚拟机中启动。这些容器像任何其他容器一样进行管理和监控,提供故障可见性以及自动重新启动这些实例的方法。
Deploying the heavyweight cluster using the CMS has many benefits. The master nodes, worker nodes, and Zookeeper (if applicable) are brought up within their own container or virtual machines. These containers are managed and monitored like any other container, providing visibility into failures as well as the means to automatically restart these instances.
您可以强制执行主节点和任何其他需要高可用性的服务的静态分配,以防止 CMS 在扩展底层计算资源时对它们进行调整。这可以防止集群监视器发出有关缺少主节点的过多警报。
You can enforce static assignment of master nodes and any other services that you require to be highly available, to prevent the CMS from shuffling them around as it scales the underlying compute resources. This prevents excessive alerts from the cluster monitor about missing master nodes.
从历史上看,重量级集群一直负责为每个提交的应用程序分配和管理资源。近年来推出的 CMS 为您提供了一个可以执行相同操作的选项,但也可以管理所有其他微服务实现。当重量级集群需要更多资源进行扩展时,必须首先向CMS请求并获取资源。然后可以将它们添加到集群的资源池中,并最终根据需要分配给应用程序。
Historically, the heavyweight cluster has been responsible for assigning and managing resources for each submitted application. The introduction of the CMS in recent years gives you an option that can do the same thing, but that can also manage all of other microservice implementations. When the heavyweight cluster requires more resources to scale up, it must first request and obtain the resources from the CMS. These can then be added to the cluster’s pool of resources and finally assigned as needed to the applications.
Spark 和 Flink 使您能够直接利用 Kubernetes 进行可扩展的应用程序部署,超越其原始的专用集群配置,其中每个应用程序都有自己的一组专用工作节点。例如,Apache Flink 允许应用程序使用 Kubernetes 在自己的隔离会话集群中独立运行。Apache Spark 提供了类似的选项,允许 Kubernetes 扮演主节点的角色,并为每个应用程序维护独立的工作资源。图 11-3显示了其工作原理的基本概述。
Spark and Flink enable you to directly leverage Kubernetes for scalable application deployment beyond their original dedicated cluster configuration, where each application has its own set of dedicated worker nodes. For example, Apache Flink allows for applications to run independently within their own isolated session cluster using Kubernetes. Apache Spark offers a similar option, allowing Kubernetes to play the role of the master node and maintain isolated worker resources for each application. A basic overview of how this works is shown in Figure 11-3.
这种部署模式与部署非重量级微服务的方式几乎相同,并融合了轻量级和 BPC 部署策略。
This deployment mode is nearly identical to how you would deploy non-heavyweight microservices and merges lightweight and BPC deployment strategies.
这种部署模式有几个优点:
There are several advantages to this deployment pattern:
它利用 CMS 资源获取模型,包括扩展需求。
It leverages the CMS resource acquisition model, including scaling needs.
工作之间是完全隔离的。
There is complete isolation between jobs.
您可以使用不同的框架和版本。
You can use different frameworks and versions.
重量级流应用程序可以像微服务一样对待,并使用相同的部署流程。
Heavyweight streaming applications can be treated just like microservices and use the same deployment processes.
当然,也有几个缺点:
And of course, there are also several disadvantages:
应用程序可以通过两种主要方式之一提交到重量级集群进行处理:驱动程序模式和集群模式。
Applications can be submitted to the heavyweight cluster for processing in one of two main ways: driver mode and cluster mode.
Spark和Flink都支持Driver模式。驱动程序只是一个单一的、本地的、独立的应用程序,帮助协调和执行应用程序,尽管应用程序本身仍然在集群资源中执行。驱动程序与集群协调以确保应用程序的进度,并可用于报告错误、执行日志记录和完成其他操作。值得注意的是,驱动程序的终止将导致应用程序的终止,这提供了一种用于部署和终止重量级流应用程序的简单机制。应用程序驱动程序可以使用CMS部署为微服务,并且可以从重量级集群获取worker资源。要终止驱动程序,只需停止它,就像它是任何其他微服务一样。
Driver mode is supported by Spark and Flink. The driver is simply a single, local, standalone application that helps coordinate and execute the application, though the application itself is still executed within the cluster resources. The driver coordinates with the cluster to ensure the progress of the application and can be used to report on errors, perform logging, and complete other operations. Notably, termination of the driver will result in termination of the application, which provides a simple mechanism for deploying and terminating heavyweight streaming applications. The application driver can be deployed as a microservice using the CMS, and the worker resources can be acquired from the heavyweight cluster. To terminate the driver, simply halt it as if it were any other microservice.
集群模式受 Spark 和 Flink 支持,是 Storm 和 Heron 作业的默认部署模式。在集群模式下,整个应用程序被提交到集群进行管理和执行,然后将唯一的ID返回给调用函数。这个唯一的 ID 对于识别应用程序并通过集群的 API 向其发出命令是必需的。使用这种部署模式,命令必须直接传递到集群来部署和停止应用程序,这可能不适合您的微服务部署管道。
Cluster mode is supported by Spark and Flink and is the default mode of deployment for Storm and Heron jobs. In cluster mode, the entire application is submitted to the cluster for management and execution, whereupon a unique ID is returned to the calling function. This unique ID is necessary for identifying the application and issuing orders to it through the cluster’s API. With this deployment mode, commands must be directly communicated to the cluster to deploy and halt applications, which may not be suitable for your microservice deployment pipeline.
有状态操作可以使用内部或外部状态来持久化(第 7 章),尽管大多数重量级框架因其高性能和可扩展性而青睐内部状态。有状态记录保存在内存中以便快速访问,但出于数据持久性目的以及当状态超出可用内存时也会溢出到磁盘。使用内部状态确实会带来一些风险,例如由于磁盘故障、节点故障导致的状态丢失以及由于 CMS 的激进扩展而导致的临时状态中断。然而,性能收益往往远远超过潜在风险,可以通过仔细规划来减轻风险。
Stateful operations may be persisted using either internal or external state (Chapter 7), though most heavyweight frameworks favor internal state for its high performance and scalability. Stateful records are kept in memory for fast access, but are also spilled to disk for data durability purposes and when state grows beyond available memory. Using internal state does carry some risks, such as state loss due to disk failure, node failures, and temporary state outages due to aggressive scaling by the CMS. However, the performance gains tend to far outweigh the potential risks, which can be mitigated with careful planning.
检查点是应用程序当前内部状态的快照,用于在扩展或节点故障后重建状态。检查点被持久保存到应用程序工作节点外部的持久存储中,以防止数据丢失。检查点可以使用与框架兼容的任何类型的存储来完成,例如 Hadoop 分布式文件系统(HDFS,常见选项)或高度可用的外部数据存储。然后,每个分区状态存储都可以从检查点恢复自身,在应用程序完全故障的情况下提供完整恢复功能,在扩展和工作节点故障的情况下提供部分恢复功能。
Checkpoints, snapshots of the application’s current internal state, are used to rebuild state after scaling or node failures. A checkpoint is persisted to durable storage external to the application worker nodes to guard against data loss. Checkpointing can be done using any sort of store that is compatible with the framework, such as Hadoop Distributed File System (HDFS, a common option) or a highly available external data store. Each partitioned state store can then restore itself from the checkpoint, providing both full restoration capabilities in the case of a total application failure, and partial restoration capabilities in the case of scaling and worker node failures.
在使用和处理分区事件流时,检查点机制必须考虑两种主要状态:
There are two main states that the checkpointing mechanism must consider when consuming and processing partitioned event streams:
partitionId< , >对offset。检查点必须确保内部键状态(参见下一项)与每个分区的消费者偏移量匹配。每个partitionId输入主题在所有输入主题中都是唯一的。
The pairs of <partitionId, offset>. The checkpoint must ensure that the internal key state (see next item) matches up with the consumer offsets of each partition. Each partitionId is unique among all input topics.
key< , >对state。这是与键控实体相关的状态,例如在聚合、缩减、窗口、连接和其他有状态操作期间。
The pairs of <key, state>. This is the state pertaining to a keyed entity, such as during aggregations, reductions, windowing, joins, and other stateful operations.
操作符和键控状态都必须同步记录,以便键控状态准确地表示标记为由操作符状态消耗的所有事件的处理。如果不这样做,可能会导致事件根本不被处理或被多次处理。图 11-4显示了记录到检查点的这种状态的示例。
Both the operator and keyed state must be synchronously recorded such that the keyed state accurately represents the processing of all the events marked as consumed by the operator state. A failure to do so may result in events either not being processed at all or being processed multiple times. An example of this state as recorded into a checkpoint is shown in Figure 11-4.
从检查点状态恢复在功能上等同于使用快照恢复外部状态存储,如“使用快照”中所述。
Restoring from a checkpointed state is functionally equivalent to using snapshots to restore external state stores, as covered in “Using snapshots”.
在处理任何新数据之前,必须从检查点完全加载与应用程序任务关联的状态。重量级框架还必须验证每个任务的操作符状态和关联的键控状态是否匹配,以确保任务之间的分区分配正确。本章开头讨论的每个主要重量级框架都以自己的方式实现检查点,因此请务必检查相应的文档以了解详细信息。
The state associated with the application task must be completely loaded from the checkpoint before you can process any new data. The heavyweight framework must also verify that the operator state and the associated keyed state match for each task, ensuring the correct assignment of partitions among tasks. Each of the major heavyweight frameworks discussed at the start of this chapter implements checkpoints in its own way, so be sure to check the corresponding documentation for the particulars.
重量级应用程序的最大并行度受到第 5 章中讨论的相同因素的限制。典型的有状态流处理器将受到最低分区流的输入计数的限制。由于重量级处理框架特别适合计算大量用户生成的数据,因此很常见的是,白天出现大量计算需求而在半夜出现很少计算需求的循环模式是很常见的。图 11-5显示了每日循环模式的示例。
The maximum parallelism of a heavyweight application is constrained by the same factors discussed in Chapter 5. A typical stateful stream processor will be limited by the input count of the lowest partitioned stream. Because heavyweight processing frameworks are particularly well suited for computing massive amounts of user-generated data, it is quite common to see cyclical patterns with significant computational requirements during the day and very few in the middle of the night. An example of a daily cyclical pattern is shown in Figure 11-5.
处理此类数据的应用程序能够随着需求的增加而扩展,并随着需求的减少而缩小,从而大大受益。适当的扩展可以确保应用程序有足够的能力及时处理所有事件,而不会因过度配置而浪费资源。理想情况下,接收事件和完全处理事件之间的延迟应该最小化,尽管许多应用程序对暂时增加的延迟并不那么敏感。
Applications that process such data benefit greatly from the ability to scale up with increasing demand and down with decreasing demand. Proper scaling can ensure that the application has sufficient capacity to process all events in a timely manner, without wasting resources by overprovisioning. Ideally, the latency between when an event is received and when it is fully processed should be minimized, though many applications are not that sensitive to temporarily increased latency.
扩展应用程序与扩展集群是分开的。这里讨论的所有扩展都假设有足够的集群资源来增加应用程序的并行性。有关集群资源扩展的信息,请参阅框架的文档。
Scaling an application is separate from scaling a cluster. All scaling discussed here assumes that there are sufficient cluster resources to increase parallelism for the application. Refer to your framework’s documentation for scaling of cluster resources.
无状态流应用程序很容易扩展或缩小。应用程序的新处理资源可以简单地加入或离开消费者组,然后重新平衡资源并恢复流式传输。有状态的应用程序可能更难处理;不仅需要将状态加载到分配给应用程序的工作线程中,而且加载的状态需要与输入事件流分区分配相匹配。
Stateless streaming applications are very easily scaled up or down. New processing resources for an application can simply join or leave the consumer group, upon which resources are rebalanced and streaming is resumed. Stateful applications can be more difficult to handle; not only does state need to be loaded into the workers assigned to the application, but the loaded state needs to match the input event stream partition assignments.
扩展有状态应用程序有两种主要策略,虽然具体情况因技术而异,但它们都有一个共同的目标,即最大限度地减少应用程序停机时间。
There are two main strategies for scaling stateful applications, and while the specifics vary depending on the technology, they share a common goal of minimizing application downtime.
第一个策略允许您删除、添加或重新分配应用程序实例,而无需停止应用程序或影响处理准确性。它仅在一些重量级流框架中可用,因为它需要仔细处理状态和随机事件。添加和删除实例需要重新分配任何分配的流分区并从最后一个检查点重新加载状态。图 11-6显示了常规的 shuffle,其中每个下游的reduce 操作都从上游groupByKey操作中获取其经过混洗的事件。如果其中一个实例突然终止,Reduce 节点将不再知道从何处获取混洗事件,从而导致致命异常。
The first strategy allows you to remove, add, or reassign application instances without stopping the application or affecting processing accuracy. It is available only in some heavyweight streaming frameworks, as it requires careful handling of both state and shuffled events. The addition and removal of instances requires redistributing any assigned stream partitions and reloading state from the last checkpoint. Figure 11-6 shows a regular shuffle, where each downstream reduce operation sources its shuffled events from the upstream groupByKey operations. If one of the instances were abruptly terminated, the reduce nodes would no longer know where to source the shuffled events from, leading to a fatal exception.
Spark 的动态资源分配实现了这种扩展策略。但需要采用粗粒度模式进行集群部署,并使用外部shuffle服务(ESS)作为隔离层。ESS接收上游任务打乱后的事件,并存储起来供下游任务消费,如图11-7所示。下游消费者通过向 ESS 询问分配给他们的数据来访问事件。
Spark’s dynamic resource allocation implements this scaling strategy. However, it requires using coarse-grained mode for cluster deployment and using an external shuffle service (ESS) as an isolation layer. The ESS receives the shuffled events from the upstream tasks and stores them for consumption by the downstream tasks, as shown in Figure 11-7. The downstream consumers access the events by asking the ESS for the data that is assigned to them.
现在可以终止任务的执行器/实例,因为下游操作不再依赖于特定的上游实例。混洗后的数据保留在ESS中,缩减后的服务可以恢复处理,如图11-8所示。在此示例中,实例 0 是唯一剩余的处理器,并承担两个分区,而下游操作通过与 ESS 的接口无缝地继续处理。
Executor/instances of tasks can now be terminated, since the downstream operations are no longer dependent on a specific upstream instance. The shuffled data remains within the ESS, and a scaled-down service, as shown in Figure 11-8, can resume processing. In this example, instance 0 is the only remaining processor and takes on both partitions, while the downstream operations seamlessly continue processing via the interface with the ESS.
实时事件流中的随机播放仍然是重量级框架的开发领域。在下一章中,我们将看看轻量级框架如何直接利用事件代理来扮演外部shuffle服务的角色。
Shuffles in real-time event stream continue to be an area of development for heavyweight frameworks. In the next chapter, we take a look at how lightweight frameworks directly leverage the event broker to play the role of the external shuffle service.
Google Dataflow 执行使用 Beam API 编写的应用程序,提供资源和工作实例的内置扩展。Heron 提供了一个(目前处于实验阶段)健康管理器,可以使拓扑动态且自我调节。此功能仍在开发中,但旨在实现实时、有状态的拓扑扩展。
Google Dataflow, which executes applications written with the Beam API, provides built-in scaling of both resources and worker instances. Heron provides a (currently experimental) Health Manager that can make a topology dynamic and self-regulating. This feature is still under development but is meant to enable real-time, stateful scaling of topologies.
第二种策略是通过重新启动应用程序来扩展应用程序,所有重量级流框架都支持这种策略。流的消耗被暂停,应用程序被检查点,然后被停止。接下来,使用新资源和并行性重新初始化应用程序,并根据需要从检查点重新加载状态数据。例如, Flink为此提供了一个简单的 REST 机制,而Storm 则提供了自己的重新平衡命令。
The second strategy, scaling an application by restarting it, is supported by all heavyweight streaming frameworks. Consumption of streams is paused, the application is checkpointed, and then it is stopped. Next, the application is reinitialized with the new resources and parallelism, with stateful data reloaded from the checkpoints as required. For example, Flink provides a simple REST mechanism for these purposes, while Storm provides its own rebalance command.
自动扩展是根据特定指标自动扩展应用程序的过程。这些指标可能包括处理延迟、消费者延迟、内存使用率和 CPU 使用率等。某些框架可能内置了自动缩放选项,例如 Google 的 Dataflow 引擎、Heron 的 Health Manager 和 Spark Streaming 的动态分配功能。其他人可能会要求您收集自己的性能和资源利用率指标,并将它们连接到框架的扩展机制,例如“消费者偏移延迟监控”中讨论的延迟监控工具。
Autoscaling is the process of automatically scaling applications in response to specific metrics. These metrics may include processing latency, consumer lag, memory usage, and CPU usage, to name a few. Some frameworks may have autoscaling options built in, such as Google’s Dataflow engine, Heron’s Health Manager, and Spark Streaming’s dynamic allocation functionality. Others may require you to collect your own performance and resource utilization metrics and wire them up to the scaling mechanism of your framework, such as the lag monitor tooling discussed in “Consumer Offset Lag Monitoring”.
重量级集群被设计为能够高度容忍长时间运行的作业不可避免的失败。主节点、工作节点和 Zookeeper 节点(如果适用)的故障都可以得到缓解,以允许应用程序几乎不间断地继续运行。这些容错功能内置于集群框架中,但可能需要您在部署集群时配置其他步骤。
Heavyweight clusters are designed to be highly tolerant to the inevitable failures of long-running jobs. Failures of the master nodes, worker nodes, and Zookeeper nodes (if applicable) can all be mitigated to allow applications to continue virtually uninterrupted. These fault-tolerance features are built into the cluster framework, but can require you to configure additional steps when deploying your cluster.
如果工作节点发生故障,正在该节点上执行的任务将转移到另一个可用的工作节点。任何所需的内部状态都会从最近的检查点与分区分配一起重新加载。主节点故障对于已经执行的应用程序应该是透明的,但根据集群的配置,您可能无法在主节点中断期间部署新作业。Zookeeper(或类似技术)支持的高可用性模式可以减轻主节点的丢失。
In the case of a worker node failure, the tasks that were being executed on that node are moved to another available worker. Any required internal state is reloaded from the most recent checkpoint along with the partition assignments. Master node failures should be transparent to applications already being executed, but depending on your cluster’s configuration you may be unable to deploy new jobs during a master node outage. High-availability mode backed by Zookeeper (or similar technology) can mitigate the loss of a master node.
确保您对主节点和工作节点有适当的监控和警报。虽然单个集群节点故障不一定会停止处理,但它仍然会降低性能并阻止应用程序从连续故障中恢复。
Make sure you have proper monitoring and alerting for your master and worker nodes. While a single cluster node failure won’t necessarily halt processing, it can still degrade performance and prevent applications from recovering from successive failures.
除了集群管理的开销之外,随着给定集群上应用程序数量的增长,您还必须考虑多租户问题。具体来说,您应该考虑资源获取的优先级、备用资源与已提交资源的比率以及应用程序可以声明资源的速率(即扩展)。例如,从输入主题开始的新流应用程序可能会请求并获取大部分空闲集群资源,从而限制任何当前正在运行的应用程序获取自己的资源。这可能会导致应用程序无法实现其服务级别目标 (SLO) 并造成下游业务问题。
Aside from the overhead of cluster management, you must account for multitenancy issues as the number of applications on a given cluster grows. Specifically, you should consider the priority of resource acquisition, the ratio of spare to committed resources, and the rate at which applications can claim resources (i.e., scaling). For instance, a new streaming application starting from the beginning of time on its input topics may request and acquire the majority of free cluster resources, restricting any currently running applications from acquiring their own. This can cause applications to miss their service-level objectives (SLOs) and create downstream business issues.
以下是缓解这些挑战的几种方法:
Here are a couple of methods to mitigate these challenges:
每个团队或业务部门都可以拥有自己的集群,并且这些集群可以彼此完全分开。当您可以通过内部开发工作或使用第三方服务提供商以编程方式申请集群以保持较低的运营开销时,此方法效果最佳。由于运行集群的开销(在协调节点(例如,主节点和 Zookeeper 节点)和监视/管理集群方面),这种方法可能会产生更高的财务成本。
Each team or business unit can have its own cluster, and these can be kept fully separate from one another. This approach works best when you can requisition clusters programmatically to keep operational overhead low, either through in-house development work or by using a third-party service provider. This approach may incur higher financial costs due to the overhead of running a cluster, both in terms of coordinating nodes (e.g., master and Zookeeper nodes) and monitoring/managing the clusters.
单个集群可以划分为具有特定资源分配的命名空间。每个团队或业务组都可以在自己的命名空间内分配自己的资源。在该命名空间内执行的应用程序只能获取这些资源,从而防止它们通过积极获取而导致命名空间之外的应用程序挨饿。此选项的缺点是,即使不需要,也必须将备用资源分配给每个命名空间,这可能会导致未使用资源的碎片池更大。
A single cluster can be divided into namespaces with specific resource allocation. Each team or business group can be assigned its own resources within their own namespace. Applications executed within that namespace can acquire only those resources, preventing them from starving applications outside of the namespace through aggressive acquisition. A downside to this option is that spare resources must be allocated to each namespace even when they’re not needed, potentially leading to a larger fragmented pool of unused resources.
重量级流处理框架植根于其前辈的 JVM 语言,其中 Java 最为常见,其次是 Scala。Python 也很常见,因为它是数据科学家和机器学习专家中流行的语言,这些专家占这些框架传统用户的很大一部分。MapReduce 风格的 API 很常用,其中的操作作为数据集上的不可变操作链接在一起。重量级框架对其 API 支持的语言有相当的限制。
Heavyweight stream processing frameworks are rooted in the JVM languages of their predecessors, with Java being the most common, followed by Scala. Python is also commonly represented, as it is a popular language among data scientists and machine learning specialists, who make up a large portion of these frameworks’ traditional users. MapReduce-style APIs are commonly used, where operations are chained together as immutable operations on data sets. Heavyweight frameworks are fairly restrictive in the languages their APIs support.
类似 SQL 的语言也变得越来越普遍。这些允许用 SQL 转换来表达拓扑,并减少学习新框架的特定 API 的认知开销。Spark、 Flink、Storm和 Beam都提供类似 SQL 的语言,尽管它们在功能和语法上有所不同,并且并非支持所有操作。
SQL-like languages are also becoming more common. These allow for topologies to be expressed in terms of SQL transformations, and reduce the cognitive overhead of learning the specific API for a new framework. Spark, Flink, Storm, and Beam all provide SQL-like languages, though they differ in features and syntax and not all operations are supported.
选择重量级流处理框架与选择 CMS 和事件代理非常相似。您必须确定您的组织愿意授权多少运营开销,以及该支持是否足以大规模运行完整的生产集群。此开销包括常规运营职责,例如监控、扩展、故障排除、调试和分配成本,所有这些都是实施和部署实际应用程序的外围内容。
Choosing a heavyweight stream-processing framework is much like selecting a CMS and event broker. You must determine how much operational overhead your organization is willing to authorize, and if that support is sufficient for running a full production cluster at scale. This overhead includes regular operational duties such as monitoring, scaling, troubleshooting, debugging, and assigning costs, all of which are peripheral to implementing and deploying the actual applications.
软件服务提供商可能会提供这些平台作为服务,但这些选项往往比为 CMS 和事件代理选择提供商更为有限。评估您可用的选项并做出相应的选择。
Software service providers may offer these platforms as a service, though the options tend to be more limited than selecting providers for your CMS and event broker. Evaluate the options available to you and choose accordingly.
最后,框架的受欢迎程度将影响您的决定。Spark 非常受欢迎,Flink 和 Storm 不太受欢迎,但仍然被积极使用。可以通过 Apache Beam 独立于重量级框架运行时执行来编写应用程序,尽管这可能对您的组织没有用处或不关心。Heron 是 Storm 的修订版,提供了更高级的功能,但似乎是最不受欢迎的选项。将您在选择 CMS 和事件代理时所考虑的相同因素应用于选择或放弃重量级框架。
Lastly, the popularity of a framework will inform your decision. Spark is extremely popular, with Flink and Storm being less popular but still actively used. Applications can be written independently of heavyweight framework runtime execution through Apache Beam, though this may not be of use or concern to your organization. Heron, a revised form of Storm that offers more advanced features, appears to be the least popular of the options. Apply the same considerations you gave to the selection of your CMS and event broker to the selection of, or abstention from, a heavyweight framework.
请记住,重量级流框架并不能合理地实现所有事件驱动的微服务。在提交之前,请验证它是否是适合您的问题空间的正确解决方案。
Keep in mind that a heavyweight streaming framework is not reasonably capable of implementing all event-driven microservices. Verify that it is the correct solution for your problem space before committing to it.
想象一下您正在经营一家简单的在线广告公司。您通过互联网购买广告空间并将其转售给您自己的客户。这些客户希望看到他们的投资回报,在这种情况下,投资回报是通过看到广告的用户的点击率来衡量的。此外,客户可以按每次会话计费,会话定义为连续的用户活动,休息时间不超过 30 分钟。
Imagine that you are running a simple online advertising company. You purchase ad space across the internet and resell it to your own customers. These customers want to see their return on investment, which in this case is measured by the click-through rate of users who are shown an advertisement. Additionally, customers can be billed on a per-session basis, with a session defined as continuous user activity with breaks no longer than 30 minutes.
在此示例中,有两个事件流:用户广告查看和用户广告点击。目标是将这两个流聚合到会话窗口中,并在30 分钟的事件时间(不是挂钟时间)过去且用户没有执行任何新操作的情况下发出它们。有关流时间和水印的回顾,请参阅第 6 章。
In this example there are two event streams: user advertisement views and user advertisement clicks. The goal is to aggregate these two streams into session windows and emit them once an event time (not wall-clock time) of 30 minutes has passed without the user performing any new actions. Refer to Chapter 6 for a refresher on stream time and watermarks.
通常,在收集这些行为事件时,您可能会在值字段中看到其他信息,例如广告的发布位置、用户的 Web 浏览器或设备信息,或者其他各种上下文或元数据。在本示例中,视图和单击事件流都已简化为以下基本架构格式:
Normally when collecting these behavioral events, you could expect to see additional information in the value field, such as where the advertisement was published, the user’s web browser or device information, or other various contexts or metadata. For the sake of this example, both the view and click event streams have been simplified down into this basic schema format:
| 钥匙 | 价值 | 时间戳 |
|---|---|---|
|
|
(事件创建的当地时间) (the local time the event was created) |
您需要执行以下操作:
You need to perform the following operations:
将所有键分组在一起,以便给定用户的所有事件都是处理实例的本地事件。
Group all of the keys together, such that all events for a given user are local to a processing instance.
使用具有 30 分钟超时时间的窗口将事件聚合在一起。
Aggregate the events together using a window with a 30-minute timeout.
一旦达到 30 分钟限制,就发出事件窗口。
Emit the window of events once the 30-minute limit has been reached.
输出流遵循以下格式:
The output stream adheres to the following format:
| 钥匙 | 价值 |
|---|---|
|
|
该Window对象指示窗口开始的时间和结束的时间。这是复合密钥的一部分,因为随着时间的推移,用户将拥有多个会话窗口,并且会话窗口可能在用户之间重复。该组合键确保了唯一性。值中的对象Action数组用于按顺序存储操作,并允许微服务计算哪些广告视图导致其用户进行计费点击。该类Action可以表示为:
The Window object indicates the time that the window started and the time that it ended. This is part of the composite key, as users will have multiple session windows over time, and session windows may be duplicated between users. This composite key ensures uniqueness. The Action object array in the value is used to store the actions in sequential order and permits the microservice to calculate which advertisement views lead its users to billable click-throughs. This Action class can be represented as:
Action{LongeventTime;LongadvertisementId;Enumaction;//one of Click, View}
Action{LongeventTime;LongadvertisementId;Enumaction;//one of Click, View}
这个简短的 Apache Flink 源代码使用其 MapReduce 风格的 API 显示了拓扑:
This abbreviated Apache Flink source code shows the topology using its MapReduce-style API:
DataStreamclickStream=...//Create stream of click eventsDataStreamviewStream=...//Create stream of view eventsclickStream.union(viewStream).keyBy(<keyselector>).window(EventTimeSessionWindows.withGap(Time.minutes(30))).aggregate(<aggregatorfunction>).addSink(<producertooutputstream>)
DataStreamclickStream=...//Create stream of click eventsDataStreamviewStream=...//Create stream of view eventsclickStream.union(viewStream).keyBy(<keyselector>).window(EventTimeSessionWindows.withGap(Time.minutes(30))).aggregate(<aggregatorfunction>).addSink(<producertooutputstream>)
该拓扑的直观表示如图 11-9所示,并行度为 2(注意 2 个单独的实例)。
A visual representation of this topology is shown in Figure 11-9, with a parallelism of 2 (note the 2 separate instances).
每个实例的执行器都被分配了各自的任务,而这些任务又被分配了输入事件流分区以进行处理。单击流和查看流都合并为单个流逻辑流,然后按键分组userId。
The executors for each instance are assigned their tasks, which are in turn assigned the input event stream partitions for processing. Both the click and view streams are unioned into a single stream logical stream, and then grouped by the userId key.
操作符与下游和操作符keyBy一起,需要将现在合并的事件洗牌到正确的下游实例。给定键的所有事件都被消耗到同一个实例中,为其余操作提供必要的数据局部性。windowaggregate
The keyBy operator, in conjunction with the downstream window and aggregate operators, requires shuffling the now-merged events to the correct downstream instances. All events for a given key are consumed into the same instance, providing the necessary data locality for the remaining operations.
现在可以为每个用户生成会话窗口,因为每个用户的事件都是单个实例的本地事件。事件按顺序时间戳顺序添加到本地状态存储中,并对每个事件应用聚合函数,直到检测到 30 分钟或更长时间的中断。此时,事件存储将逐出已完成的会话并清除<windowId,userId>键和值的内存。
Session windows for each user can be generated now that each user’s events are local to a single instance. Events are added to the local state store in sequential timestamp order, with the aggregation function applied to each event, until a break of 30 minutes or more is detected. At this point the event store evicts the completed session and purges the memory of the <windowId,userId> key and value.
您的框架可能允许对窗口和基于时间的聚合进行额外的控制。这可以包括保留已关闭一段时间的会话和窗口,以便可以应用迟到的事件并将更新发送到输出流。检查框架的文档以获取更多信息。
Your framework may allow for additional control over windowing and time-based aggregations. This can include retaining sessions and windows that have closed for a period of time, so that late-arriving events can be applied and an update emitted to the output stream. Check the documentation of your framework for more information.
接下来,图 11-10说明了缩小到单一并行度的效果。假设没有动态缩放,您需要先停止流处理器,然后再使用新的并行度设置从检查点恢复它。启动后,该服务会从最后一个已知的良好检查点读取有状态的键控数据,并将操作员状态恢复到分配的分区。一旦状态恢复,服务就可以恢复正常的流处理。
Next, Figure 11-10 illustrates the effects of scaling down to just a single degree of parallelism. Assuming no dynamic scaling, you would need to halt the stream processor before restoring it from a checkpoint with the new parallelism setting. Upon startup, the service reads the stateful keyed data back from the last known good checkpoint and restores the operator state to the assigned partitions. Once state is restored, the service can resume normal stream processing.
第 1 阶段的运行方式与以前一样,但在这种情况下,所有分区都被分配给实例 0 中的任务使用。尽管源和目标与第 2 阶段中所示的实例保持相同,但仍会执行分组和洗牌。请记住尽管这里的所有通信完全是本地的,但在实例 0 上运行的各个任务都必须消耗其分配的随机事件。拓扑的最后一个阶段,即第 3 阶段,正常地窗口和聚合事件。
Stage 1 operates as before, though in this case all of the partitions are assigned for consumption by tasks in instance 0. Grouping and shuffling are still performed, though the source and destination remain the same instance as seen in stage 2. Keep in mind that the individual tasks running on instance 0 must each consume their assigned shuffled events, though all communication here is entirely local. The last stage of the topology, stage 3, windows and aggregates the events as normal.
本章介绍了重量级流处理框架,包括它们的发展简史以及它们旨在帮助解决的问题。这些系统具有高度可扩展性,允许您根据各种分析模式处理流,但它们可能不足以满足某些有状态事件驱动的微服务应用程序模式的要求。
This chapter introduced heavyweight stream processing frameworks, including a brief history of their development and the problems they were created to help solve. These systems are highly scalable and allow you to process streams according to a variety of analytical patterns, but they may not be sufficient for the requirements of some stateful event-driven microservice application patterns.
重量级框架使用集中式资源集群进行操作,这可能需要额外的操作开销、监控和协调才能成功集成到微服务框架中。集群和应用程序部署模型的最新创新提供了与 Kubernetes 等容器管理解决方案的更好集成,允许更精细地部署重量级流处理器,类似于完全独立的微服务。
Heavyweight frameworks operate using centralized resource clusters, which may require additional operational overhead, monitoring, and coordination to integrate successfully into a microservice framework. Recent innovations in cluster and application deployment models have provided better integration with container management solutions such as Kubernetes, allowing for more granular deployment of heavyweight stream processors similar to that of fully independent microservices.
轻量级框架提供与重量级框架类似的功能,但在某种程度上充分利用了事件代理和容器管理系统 (CMS)。与重量级框架不同,轻量级框架没有额外的专用资源集群来管理特定于框架的资源。水平扩展、状态管理和故障恢复由事件代理和 CMS 提供。应用程序作为单独的微服务部署,就像部署任何 BPC 微服务一样。并行性由消费者组成员身份和分区所有权控制。当新的应用程序实例加入和离开消费者组时,分区会被重新分配,包括共同分区的分配。
Lightweight frameworks provide similar functionality to heavyweight frameworks, but in a way that heavily leverages the event broker and the container management system (CMS). Unlike heavyweight frameworks, lightweight frameworks have no additional dedicated resource cluster for managing framework-specific resources. Horizonal scaling, state management, and failure recovery are provided by the event broker and the CMS. Applications are deployed as individual microservices, just as any BPC microservice would be deployed. Parallelism is controlled by consumer group membership and partition ownership. Partitions are redistributed as new application instances join and leave the consumer group, including copartitioned assignments.
轻量级框架提供的流处理功能可以与重量级框架相媲美,并且在许多情况下甚至超过它们。将流具体化为表,加上简单的开箱即用连接功能,可以轻松处理流以及不可避免地出现在其中的关系数据。请注意,虽然表具体化功能并不是轻量级框架所独有的,但其现成的包含性和易用性表明轻量级框架可以解决复杂的有状态问题。
Lightweight frameworks offer stream processing features that rival those of heavyweight frameworks, and in a number of cases, exceed them. Materialization of streams into tables, along with simple out-of-the-box join functionality, makes it easy to handle streams and the relational data that inevitably ends up in them. Note that while table materializing functionality is not unique to lightweight frameworks, its ready inclusion and ease of use are indicative of the complex stateful problems that lightweight frameworks can address.
轻量级模型依赖于事件代理通过使用内部事件流来提供数据局部性和共同分区的机制。事件代理还通过使用变更日志充当微服务内部状态的持久存储机制,如“将状态记录到变更日志事件流”中所述。通过利用 CMS,您可以像任何其他事件驱动应用程序一样部署轻量级微服务。您只需添加和删除实例即可处理应用程序并行性,并使用 CMS 提供扩展和故障管理机制。图 12-1说明了基本的轻量级模型,包括用于在实例之间进行通信的内部事件流。
The lightweight model relies upon the event broker to provide the mechanisms for data locality and copartitioning through the use of internal event streams. The event broker also acts as the mechanism of durable storage for a microservice’s internal state via the use of changelogs, as discussed in “Recording State to a Changelog Event Stream”. By leveraging the CMS, you deploy lightweight microservices like any other event-driven application. You handle application parallelism simply by adding and removing instances, using the CMS to provide the scaling and failure management mechanisms. Figure 12-1 illustrates the basic lightweight model, including an internal event stream used to communicate between instances.
轻量级框架的主要限制主要与当前可用的选项有关,本章稍后将对此进行介绍。
The main limitations for lightweight frameworks relate mostly to the currently available options, covered later in this chapter.
轻量级框架与重量级框架的处理方法密切相关。各个实例根据拓扑处理事件,事件代理提供实例间通信层以实现超出单个实例的可扩展性。
The lightweight framework closely mirrors the processing methodology of the heavyweight framework. Individual instances process events according to the topology, with the event broker providing an inter-instance communication layer for scalability beyond a single instance.
对于任何基于键的操作,相同键的数据必须是给定处理实例的本地数据,例如 ajoin或groupByKey后跟后续reduce/的操作aggregation。这些洗牌涉及通过内部事件流发送事件,并将给定键的每个事件写入单个分区(请参阅“共同分区事件流”),而不是使用直接实例到实例通信。
Data of the same key must be local to a given processing instance for any key-based operations, such as a join, or a groupByKey operation followed by a subsequent reduce/aggregation. These shuffles involve sending the events through an internal event stream, with each event of a given key written into a single partition (see “Copartitioning Event Streams”), instead of using direct instance-to-instance communication.
轻量级框架利用事件代理来提供此通信路径,并说明了轻量级应用程序与事件代理的更深入集成。将此与重量级框架进行对比,在重量级框架中,改组需要节点之间直接进行广泛的协调。当与 CMS 提供的应用程序管理选项结合使用时,轻量级框架比重量级框架更符合现代微服务所需的应用程序部署和管理。
The lightweight framework leverages the event broker to provide this communication path and illustrates the deeper integration of the lightweight application with the event broker. Contrast this with the heavyweight framework, where shuffling requires extensive coordination directly between the nodes. When combined with application management options provided by the CMS, the lightweight framework is much more aligned than the heavyweight framework with the application deployment and management required of modern-day microservices.
轻量级框架的默认操作模式是使用由事件代理中存储的更改日志支持的内部状态。使用内部状态允许每个微服务使用部署配置来控制它获取的资源。
The default mode of operation for lightweight frameworks is to use internal state backed by changelogs stored in the event broker. Using internal state allows for each microservice to control the resources it acquires using deployment configurations.
由于每个轻量级应用程序都完全独立于其他应用程序,因此一个应用程序可以请求在具有非常高性能的本地磁盘的实例上运行,而另一个应用程序可以请求在具有极大(尽管可能慢得多)硬盘驱动器的实例上运行。
Since every lightweight application is fully independent of the others, one application could request to run on instances with very high-performance local disk, while another could request to run on instances with extremely large, albeit perhaps much slower, hard-disk drives.
还可以插入不同的存储引擎,允许您使用具有替代模型和查询引擎的外部状态存储。这可以获得轻量级框架功能的所有好处,同时还支持图形数据库和文档存储等选项。
Different storage engines can also be plugged in, allowing you to use external state stores with alternative models and querying engines. This reaps all the benefits of the lightweight framework functionality, while also enabling options such as graph databases and document stores.
与重量级框架的检查点模型相反,使用内部状态存储的轻量级框架利用事件代理来存储其变更日志。这些变更日志提供了扩展和故障恢复所需的持久性。
In contrast to the checkpoint model of the heavyweight frameworks, lightweight frameworks using internal state stores leverage the event broker to store their changelogs. These changelogs provide the durability required for both scaling and failure recovery.
扩展微服务和从故障中恢复实际上是相同的过程。由于有意扩展长时间运行的进程或由于实例恢复失败,添加应用程序实例需要正确分配分区以及任何伴随状态。同样,故意或由于故障而删除实例需要将分区分配和状态重新分配给另一个活动实例,以便处理可以不间断地继续。
Scaling a microservice and recovering from failures are effectively the same process. Adding an application instance, due to intentional scaling of a long-running process or due to a failed instance recovering, requires that partitions be correctly assigned alongside any accompanying state. Similarly, removing an instance, deliberately or due to failure, requires that partition assignments and state be reassigned to another live instance so that processing can continue uninterrupted.
轻量级框架模型的主要好处之一是应用程序可以在执行时动态扩展。尽管由于消费者组重新平衡和更改日志中的状态重新实现而可能导致处理延迟,但无需重新启动应用程序来更改并行性。图 12-2说明了扩展应用程序的过程。分配的输入分区将被重新平衡(包括任何内部流),并且状态将在继续工作之前从更改日志中恢复。
One of the main benefits of the lightweight framework model is that applications can be dynamically scaled as they are under execution. There is no need to restart an application just to change parallelism, though there may be a delay in processing due to consumer group rebalancing and rematerialization of state from the changelog. Figure 12-2 illustrates the process of scaling an application up. The assigned input partitions are rebalanced (including any internal streams) and the state is restored from the changelogs prior to continuation of work.
让我们看看扩展轻量级应用程序的主要考虑因素。
Let’s look at the main considerations of scaling a lightweight application.
轻量级框架微服务中的事件混排很简单,因为事件被重新分区到内部事件流中以供下游使用。此内部事件流将创建混洗事件的上游实例与使用这些事件的下游实例隔离开来,充当类似于重量级框架执行动态扩展所需的混洗服务。任何动态扩展只需要将消费者重新分配给内部事件流,而不考虑生产者。
Event shuffling in lightweight framework microservices is simple, as events are repartitioned into an internal event stream for downstream consumption. This internal event stream isolates the upstream instances that create the shuffled events from the downstream ones that consume them, acting as a shuffle service similar to that required by heavyweight frameworks to perform dynamic scaling. Any dynamic scaling requires only that the consumers be reassigned to the internal event stream, regardless of the producers.
扩展后,具有新内部状态分配的实例必须在处理任何新事件之前从变更日志加载数据。此过程类似于重量级解决方案中从持久存储加载检查点的方式。<partitionId, offset>所有事件流分区(输入和内部)的运算符状态( 的映射)存储在各个应用程序的使用者组中。键控状态(成对<key, state>)存储在应用程序中每个状态存储的更改日志中。
Upon scaling, an instance with new internal state assignments must load the data from the changelog before processing any new events. This process is similar to how checkpoints are loaded from durable storage in heavyweight solutions. The operator state (the mappings of <partitionId, offset>) for all event stream partitions, both input and internal, is stored within the consumer group for the individual application. The keyed state (pairs of <key, state>) is stored within the changelog for each state store in the application.
从更改日志重新加载时,应用程序实例必须在处理任何新事件之前优先考虑所有内部状态数据的消耗和加载。这是状态恢复阶段,在状态完全恢复之前对事件的任何处理都有可能产生不确定的结果。一旦应用程序拓扑中每个状态存储的状态完全恢复,输入流和内部流的消耗就可以安全地恢复。
When reloading from a changelog, the application instance must prioritize consumption and loading of all internal stateful data prior to processing any new events. This is the state restoration phase, and any processing of events before state is fully restored risks creating nondeterministic results. Once state has been fully restored for each state store within the application topology, consumption of both input and internal streams may be safely resumed.
正如“使用热副本”中所介绍的,热副本是根据变更日志具体化的状态存储的副本。当提供数据的主实例发生故障时,它提供备用回退,但也可用于优雅地缩减有状态应用程序。当实例终止并且消费者组重新平衡时,可以分配分区以利用热副本的状态并继续处理而不会中断。热副本允许您在扩展和故障期间保持高可用性,但它们确实以额外的磁盘和处理器使用为代价。
A hot replica, as introduced in “Using hot replicas”, is a copy of a state store materialized off of the changelog. It provides a standby fallback for when the primary instance serving that data fails, but can also be used to gracefully scale down stateful applications. When an instance is terminated and a consumer group is rebalanced, partitions can be assigned to leverage the hot replica’s state and continue processing without interruption. Hot replicas allow you to maintain high availability during scaling and failures, but they do come at the cost of additional disk and processor usage.
同样,您可以使用热副本无缝地扩展实例数量,而不必因新节点上的状态重新物化而遭受处理暂停。轻量级框架当前面临的问题之一是扩展实例通常遵循当前的工作流程:
Similarly, you can use hot replicas to seamlessly scale up the instance count without having to suffer through processing pauses due to state rematerialization on the new node. One of the current issues facing lightweight frameworks is that scaling an instance up typically follows the current workflow:
启动一个新实例。
Start a new instance.
加入消费者组并重新平衡分区所有权。
Join the consumer group and rebalance partition ownership.
当状态从变更日志中具体化时暂停(这可能需要一些时间)。
Pause while state is materialized from the changelog (this can take some time).
恢复处理。
Resume processing.
一种选择是在新实例上填充状态副本,等到它赶上更改日志的头部,然后重新平衡以为其分配输入分区的所有权。此模式减少了由于具体化变更日志流而导致的中断,并且只需要事件代理提供额外的带宽即可实现。目前正在为 Kafka Streams 开发此功能。
One option is to populate a replica of the state on the new instance, wait until it’s caught up to the head of the changelog, and then rebalance to assign it ownership of the input partitions. This mode reduces outages due to materializing the changelog streams and only requires extra bandwidth from the event broker to do so. This functionality is currently under development for Kafka Streams.
目前,适合轻量级框架模型的主要选项有两个,这两个选项都需要使用 Apache Kafka 事件代理。这两个框架都在其高级 API 中提供无限期保留的物化流,解锁主键连接和外键连接等选项。这些联接模式允许您处理关系数据,而无需具体化到外部状态存储,从而减少编写基于联接的应用程序的认知开销。
Currently, there are two main options that fit the lightweight framework model, both of which require the use of the Apache Kafka event broker. Both frameworks provide indefinitely retained materialized streams in their high-level APIs, unlocking options such as primary-key joins and foreign-key joins. These join patterns permit you to handle relational data without having to materialize to external state stores and therefore reduce the cognitive overhead of writing join-based applications.
Kafka Streams 是一个功能丰富的流处理库,嵌入在单个应用程序中,其中输入和输出事件存储在 Kafka 集群中。它将编写和部署基于 JVM 的标准应用程序的简单性与利用 Kafka 集群深度集成的强大流处理框架相结合。
Kafka Streams is a feature-rich stream processing library that is embedded within an individual application, where the input and output events are stored in the Kafka cluster. It combines the simplicity of writing and deploying standard JVM-based applications with a powerful stream-processing framework leveraging deep integration with the Kafka cluster.
Samza 提供了许多与 Kafka Streams 相同的功能,尽管它在一些与独立部署相关的功能方面落后。Samza 早于 Kafka Streams,其最初的部署模型基于使用重量级集群。直到最近,Samza 才发布了嵌入式模式,该模式密切反映了 Kafka Streams 的应用程序编写、部署和管理生命周期。
Samza offers many of the same features as Kafka Streams, though it lags behind in some features related to independent deployment. Samza predates Kafka Streams, and its original deployment model is based on using a heavyweight cluster. It is only relatively recently that Samza released an embedded mode, which closely mirrors Kafka Streams’ application writing, deployment, and management lifecycle.
Samza 的嵌入模式允许您将此功能嵌入到各个应用程序中,就像任何其他 Java 库一样。这种部署模式不再需要专用的重量级集群,而是依赖于上一节讨论的轻量级框架模型。默认情况下,Samza 确实依赖使用 Apache Zookeeper 来跨各个实例进行协调,但您可以修改它以使用其他协调机制(例如 Kubernetes)。
Samza’s embedded mode allows you to embed this functionality within individual applications, just like any other Java library. This deployment mode removes the need for a dedicated heavyweight cluster, instead relying on the lightweight framework model discussed in the previous section. By default, Samza does rely on using Apache Zookeeper for coordination across individual instances, but you can modify this to use other coordination mechanisms such as Kubernetes.
Apache Samza 的嵌入式模式可能无法提供集群模式下的所有功能。
Apache Samza’s embedded mode may not provide all of the functionality that it has in cluster mode.
轻量级框架不像重量级框架或基本消费者/生产者模式的消费者/生产者库那么常见。轻量级框架确实广泛依赖于与事件代理的集成,这限制了它们对其他事件代理技术的可移植性。轻量级框架领域仍然相当年轻,但随着 EDM 领域的成熟,它肯定会成长和发展。
Lightweight frameworks are not as common as heavyweight frameworks or consumer/producer libraries for the basic consumer/producer pattern. Lightweight frameworks do rely extensively on integration with the event broker, which limits their portability to other event broker technologies. The lightweight framework domain is still fairly young, but is sure to grow and develop as the EDM space matures.
Kafka Streams 和 Samza 都基于 Java,这限制了它们只能用于基于 JVM 的语言。高级 API 以 MapReduce 语法的形式表示,就像重量级框架语言中的情况一样。那些对函数式编程或上一章讨论的任何重量级框架有经验的人,使用这些框架都会感到很自在。
Both Kafka Streams and Samza are based in Java, which limits their use to JVM-based languages. The high-level APIs are expressed as a form of MapReduce syntax, as is the case in the heavyweight framework languages. Those who are experienced with functional programming, or any of the heavyweight frameworks discussed in the previous chapter, will feel right at home using either of these frameworks.
Apache Samza 支持开箱即用的类 SQL 语言,尽管其功能目前仅限于简单的无状态查询。Kafka Streams 没有第一方 SQL 支持,尽管其企业赞助商Confluence在其自己的社区许可证下提供 KSQL。与重量级解决方案非常相似,这些 SQL 解决方案是底层流库之上的包装器,可能无法提供直接从流库提供的全部功能和特性。
Apache Samza supports a SQL-like language out of the box, though its functionality is currently limited to simple stateless queries. Kafka Streams doesn’t have first-party SQL support, though its enterprise sponsor, Confluent, provides KSQL under its own community license. Much like the heavyweight solutions, these SQL solutions are wrappers on top of the underlying stream libraries and may not provide the entirety of functions and features that would otherwise be available from the stream libraries directly.
假设您与“示例:点击和查看的会话窗口化”中的同一家大型广告公司工作,但您是会话窗口的下游消费者。快速提醒一下,窗口会话事件的格式如表 12-1所示。
Say you are working for the same large-scale advertising company as in “Example: Session Windowing of Clicks and Views”, but you are a downstream consumer of the session windows. As a quick reminder, the format of the windowed session events is shown in Table 12-1.
| 钥匙 | 价值 |
|---|---|
|
|
这里,一个动作构成如下:
Here, an action constitutes the following:
行动 {
事件时间长;
长广告Id;
枚举动作;//单击、查看之一
}Action {
Long eventTime;
Long advertisementId;
Enum action; //one of Click, View
}
您团队的目标是进行直播Advertisement-Sessions并执行以下操作:
Your team’s goal is to take the Advertisement-Sessions stream and do the following:
对于每个广告查看操作,确定其后是否有点击。将每个观看-点击对求和并输出为转化事件,格式如表 12-2所示:
| 钥匙 | 价值 |
|---|---|
|
|
For each advertisement view action, determine if there is a click that comes after it. Sum each view-click pair and output as a conversion event, as formatted by Table 12-2:
| Key | Value |
|---|---|
|
|
按 分组所有广告转化事件advertisementId,并将它们的值相加得出总计。
Group all of the advertisement conversion events by advertisementId, and sum their values into a grand total.
advertisementId通过物化广告实体流连接所有转化事件,以便您的服务可以确定哪个客户拥有广告以进行计费,如表 12-3所示:
| 钥匙 | 价值 |
|---|---|
|
|
Join all conversion events by advertisementId on the materialized advertisement entity stream so that your service can determine which customer owns the advertisement for billing purposes, as shown in Table 12-3:
| Key | Value |
|---|---|
|
|
以下是您可能会在该操作中看到的一些 Kafka Streams 源代码的示例。KStream是流的高级抽象,而KTable是表的高级抽象,通过具体化广告实体流而生成。
Here’s an example of some Kafka Streams source code that you could expect to see for this operation. A KStream is the high-level abstraction of a stream, while a KTable is a high-level abstraction of a table, generated by materializing the advertisement entity stream.
KStream<WindowKey,Actions>userSessions=...//Transform 1 userSession into 1 to N conversion events, rekey on AdvertisementIdKTable<AdvertisementId,Long>conversions=userSessions.transform(...)//transform userSessions into conversion events.groupByKey().aggregate(...)//Creates an aggregate KTable//Materialize the advertisement entitiesKTable<AdvertisementId,Advertisement>advertisements=...//The tables are automatically co-partitioned by including//the join operation in the topology.conversions.join(advertisements,joinFunc)//See stage 4 for more details..to("AdvertisementEngagements")
KStream<WindowKey,Actions>userSessions=...//Transform 1 userSession into 1 to N conversion events, rekey on AdvertisementIdKTable<AdvertisementId,Long>conversions=userSessions.transform(...)//transform userSessions into conversion events.groupByKey().aggregate(...)//Creates an aggregate KTable//Materialize the advertisement entitiesKTable<AdvertisementId,Advertisement>advertisements=...//The tables are automatically co-partitioned by including//the join operation in the topology.conversions.join(advertisements,joinFunc)//See stage 4 for more details..to("AdvertisementEngagements")
代码表示的拓扑如下,如图12-3所示。
The topology represented by the code is as follows, illustrated in Figure 12-3.
该Advertisement-Sessions流包含太多事件,单个实例无法处理,因此代码需要使用多个处理实例进行并行化。在此示例中,最大并行化级别最初为 3,因为这是实体Advertisements流的分区计数。在非高峰时段,可能只使用一两个实例,但在用户活动频繁期间,应用程序将落后。
幸运的是,Advertisements可以通过内部流将实体流重新分区到匹配的 12 个分区。这些事件被简单地消耗并重新分区到一个新的 12 分区内部流中。广告实体基于advertisementId来自阶段 1b 和 2b 的转化事件进行协同定位。
The Advertisement-Sessions stream contains too many events for a single instance to process, so the code needs to parallelize using multiple processing instances. In this example, the maximum level of parallelization is initially 3, as that is the partition count of the Advertisements entity stream. During off-peak hours it may be possible to just use one or two instances, but during periods of heavy user activity the application will fall behind.
Fortunately, the Advertisements entity stream can be repartitioned up to a matching 12 partitions by means of an internal stream. The events are simply consumed and repartitioned into a new 12-partition internal stream. The advertisement entities are colocated based on advertisementId with the conversion events from stages 1b and 2b.
事件从事件流中消耗Advertisement-Sessions,转换事件以表格形式列出并发出(键 = Long advertisementId,值 = Long conversionSum)。请注意,对于给定的会话事件,可以创建多个转化事件,每个转化事件对应每对查看点击事件advertisementId。advertisementId这些事件与阶段 1a 和 2a 中的广告实体位于同一位置。
Events are consumed from the Advertisement-Sessions event stream, with conversion events tabulated and emitted (key = Long advertisementId, value = Long conversionSum). Note that for a given session event there can be multiple conversion events created, one for each pair of view-click events per advertisementId. These events are colocated based on advertisementId with the advertisement entities from stages 1a and 2a.
现在必须将事件聚合成表 12-4Advertisement-Conversions中所示的物化表格式,因为企业有兴趣保留与每个实体的约定的无限期保留记录。聚合是给定 的所有值的简单总和。AdvertisementadvertisementId
| 长广告ID | 长转换和 |
|---|---|
|
|
|
|
|
|
因此,由该聚合运算符处理的新Advertisement-Conversionskey 事件将增加from到(AdKey1, 15)的内部状态存储值。AdKey1402417
The Advertisement-Conversions events must now be aggregated into the materialized table format shown in Table 12-4, since the business is interested in keeping an indefinitely retained record of the engagements with each Advertisement entity. The aggregation is a simple sum of all values for a given advertisementId.
| Long advertisementId | Long conversionSum |
|---|---|
|
|
|
|
|
|
Thus, a new Advertisement-Conversions event of key (AdKey1, 15) processed by this aggregation operator would increment the internal state store value of AdKey1 from 402 to 417.
此拓扑的最后一步是将Total-Advertisement-Conversions第 3 阶段创建的物化表与重新分区的Advertisement实体流连接起来。您已经通过在阶段 2a 和 2b 中对输入流进行共同分区来为此连接奠定了基础,确保所有advertisementId数据都位于其处理实例的本地。流中的实体Advertisement被具体化到它们自己的分区本地状态存储中,并随后与Total-Advertisement-Conversions.
The last step of this topology is to join the materialized Total-Advertisement-Conversions table created in stage 3 with the repartitioned Advertisement entity stream. You’ve already established the groundwork for this join by copartitioning the input streams in stages 2a and 2b, ensuring that all advertisementId data is local to its processing instance. The entities in the Advertisement stream are materialized into their own partition-local state stores and subsequently joined with the Total-Advertisement-Conversions.
连接函数用于指定连接所需的结果,就像selectSQL 中使用语句来仅选择应用程序所需的字段一样。此场景的 Java 连接函数可能如下所示:
A join function is used to specify the desired results from the join, much like a select statement is used in SQL to select only the fields that the application requires. A Java join function for this scenario may look something like this:
publicEnrichedAdjoinFunction(Longsum,Advertisementad){if(sum!=null||ad!=null)returnnewEnrichedAd(sum,ad.name,ad.type);else//Return a tombstone, as one of the inputs is null,//possibly indicating a deletion.returnnull;}
publicEnrichedAdjoinFunction(Longsum,Advertisementad){if(sum!=null||ad!=null)returnnewEnrichedAd(sum,ad.name,ad.type);else//Return a tombstone, as one of the inputs is null,//possibly indicating a deletion.returnnull;}
前面joinFunction假设任一输入可能为空,表示该事件的上游删除。因此,您需要确保您的代码向下游的使用者输出相应的逻辑删除。值得庆幸的是,大多数框架(轻量级和重量级)都会区分内部、左、右、外部和外键连接,并执行一些幕后操作来避免您通过连接函数传播逻辑删除。但是,出于说明目的,重要的是要考虑微服务拓扑中逻辑删除事件的影响。
The preceding joinFunction assumes that either input may be null, indicating an upstream deletion of that event. Accordingly, you need to ensure that your code outputs a corresponding tombstone downstream to its consumers. Thankfully, most frameworks (both lightweight and heavyweight) differentiate between inner, left, right, outer, and foreign-key joins, and do some behind-the-scenes operations to save you from propagating tombstones through your join functions. However, for the purposes of instruction, it’s important that you consider the effects of tombstone events in your microservice topology.
Kafka Streams 拓扑与joinFunctionSQL 中的选择表达式相同:
The Kafka Streams topology and joinFunction is identical to the selection expression in SQL:
SELECTadConversionSumTable.sum,adTable.name,adTable.typeFROMadConversionSumTableFULLOUTERJOINadTableONadConversionSumTable.id=adTable.id
SELECTadConversionSumTable.sum,adTable.name,adTable.typeFROMadConversionSumTableFULLOUTERJOINadTableONadConversionSumTable.id=adTable.id
在这种情况下,输出事件流的物化视图Enriched-Advertising-Engagements如表 12-5所示。
In this case, the materialized view of the Enriched-Advertising-Engagements output event stream looks like Table 12-5.
| 广告ID(键) | 丰富的广告(价值) |
|---|---|
|
|
|
|
|
|
|
|
|
|
此示例表显示了第 3 阶段的预期聚合,并与Advertising实体数据连接。AdKey4每个AdKey5都显示完整外连接的结果: 尚未发生任何转化AdKey4,而 尚无可用的广告实体数据AdKey5。
This sample table shows the expected aggregations from stage 3, joined with the Advertising entity data. AdKey4 and AdKey5 each show the results of a full outer join: no conversions have yet occurred for AdKey4, while there is no advertising entity data yet available for AdKey5.
本章介绍了轻量级流处理框架,包括它们的主要优点和权衡。这些是高度可扩展的处理框架,广泛依赖与事件代理的集成来执行大规模数据处理。与容器管理系统的高度集成为每个单独的微服务提供了可扩展性要求。
This chapter introduced lightweight stream processing frameworks, including their major benefits and tradeoffs. These are highly scalable processing frameworks that rely extensively on integration with the event broker to perform large-scale data processing. Heavy integration with the container management system provides the scalability requirements for each individual microservice.
与重量级框架相比,轻量级框架仍然相对较新。然而,它们提供的功能往往非常适合构建长期运行的、独立的、有状态的微服务,并且当然值得您自己的业务用例进行探索。
Lightweight frameworks are still relatively new compared to heavyweight frameworks. However, the features they provide tend to be well suited for building long-running, independent, stateful microservices, and are certainly worth exploring for your own business use cases.
尽管事件驱动的微服务模式非常强大,但它们无法满足组织的所有业务需求。请求-响应端点提供了实时提供重要数据的方法,特别是当您:
As powerful as event-driven microservice patterns are, they cannot serve all of the business needs of an organization. Request-response endpoints provide the means to serve important data in real time, particularly when you are:
从外部来源收集指标,例如用户手机或物联网 (IoT) 设备上的应用程序
Collecting metrics from external sources, such as an application on a user’s cell phone or Internet of Things (IoT) devices
与现有的请求响应应用程序集成,特别是组织外部的第三方应用程序
Integrating with existing request-response applications, particularly third-party ones outside of the organization
向网络和移动设备用户实时提供内容
Serving content in real time to web and mobile device users
根据实时信息(例如位置、时间和天气)提供动态请求
Serving dynamic requests based on real-time information, such as location, time, and weather
事件驱动模式仍然在此领域发挥着重要作用,将它们与请求响应解决方案集成将帮助您利用两者的最佳功能。
Event-driven patterns still play a large role in this domain, and integrating them with request-response solutions will help you leverage the best features of both.
就本章而言,术语“请求-响应服务”是指通常通过同步 API 直接相互通信的服务。通过 HTTP 进行通信的两个服务是请求-响应通信的一个主要示例。
For the purposes of this chapter, the term request-response services refers to services that communicate with each other directly, typically through a synchronous API. Two services communicating via HTTP is a prime example of request-response communication.
由于历史、优先级、熟悉性、便利性以及一系列其他原因,外部事件主要通过请求-响应 API 从外部发送。虽然可以将事件代理及其流公开给外部 客户端,但这样做在很大程度上是不合理的,因为您需要解决许多与访问和安全相关的问题。那很好。请求-响应 API 非常适合这些场景,就像几十年前一样 。有两种主要类型的外部生成事件需要考虑。
Due to history, precedence, familiarity, convenience, and a whole host of other reasons, external events are predominantly sent from the outside via a request-response API. While it is possible to expose an event broker and its streams to an external client, it is largely unreasonable to do so, as you would need to resolve a number of issues relating to access and security. And that is fine. Request-response APIs work wonderfully for these scenarios, just as they have for many decades before. There are two main types of externally generated events to consider.
第一类事件是由您的产品自主地从客户端发送到服务器的事件。这些请求通常被定义为产品的度量或测量,例如有关用户正在做什么的信息、活动的定期测量或某种类型的传感器读数。这些事件统称为 分析事件,描述了有关产品操作的测量结果和事实陈述(“示例:过载事件定义”显示了此类事件的实际情况)。安装在客户手机上的应用程序是外部事件源的一个很好的示例。考虑像 Netflix 这样的媒体流服务,其中分析事件可以独立发送回来以测量诸如您已经开始播放哪些电影以及您观看了多少电影等信息。来自外部产品的任何基于源自该产品的操作的请求都被视为外部生成的事件。
The first type of events are those sent from client to server autonomously by your products. These requests are usually defined as a metric or measurement from the product, such as information about what a user is doing, periodic measurements of activity, or sensor readings of some sort. Collectively known as analytical events, these describe measurements and statements of fact about the operation of the product (“Example: Overloading event definitions” shows such an event in action). An application installed on a customer’s cell phone is a good example of an external event source. Consider a media streaming service like Netflix, where analytical events can be independently sent back to measure things such as which movies you have started and how much of them you’ve watched. Any request from an external product, based on actions originating from that product, counts as an externally generated event.
现在,您可能想知道加载当前电影接下来 60 秒的请求是否算作外部生成的事件。绝对是的,他们确实这么做了。但真正要问的问题是,“这些事件对业务是否足够重要,以至于它们必须进入自己的事件流进行额外处理?” 在许多情况下,答案是否定的,您不会收集这些事件并将其存储在事件流中。但对于那些答案是肯定的情况,您可以简单地将请求解析为事件并将其路由到其自己的事件流中。
Now, you may be wondering if, say, requests to load the next 60s of the current movie count as an externally generated event. Absolutely, they do. But the real question to ask is, “Are these events important enough to the business such that they must go into their own event stream for additional processing?” In many cases the answer is no, and you would not collect and store those events in an event stream. But for those cases where the answer is yes, you can simply parse the request into an event and route it into its own event stream.
第二种类型的外部生成事件是响应事件,它是响应而生成的来自您的一项服务的请求。您的服务编写请求,将其发送到端点,然后等待响应。在某些情况下,实际上只重要的是收到请求,并且请求客户端不需要响应中的任何其他详细信息。例如,如果您需要发出发送广告电子邮件的请求,那么从处理请求的第三方服务收集响应如果转变为事件可能没有用。一旦请求成功发出(HTTP 202 响应),您就可以假设第三方电子邮件应用程序将会实现这一点。如果无需根据结果采取任何操作,则可能不需要收集响应并将其转换为事件。
The second type of externally generated event is a reactive event, which is generated in response to a request from one of your services. Your service composes a request, sends it to the endpoint, and awaits a response. In some cases, it’s really only important that the request is received, and the requesting client doesn’t need any other details from the response. For example, if you need to issue requests to send advertisement emails, collecting the response from the third-party service handling the requests may not be useful if turned into events. Once the request is successfully issued (HTTP 202 response), you can assume that the third-party email application will make it happen. Collecting the responses and converting them into events may not be necessary if there is no action to take from the results.
另一方面,您的业务需求可能期望从请求的响应中获得重要的细节。一个典型的例子是使用第三方支付服务,其中输入事件说明客户应支付的金额。来自第三方 API 的响应负载非常重要,因为它指定支付是否成功、任何错误消息以及任何其他详细信息,例如指示支付信息的唯一的、可追踪的号码。这些数据对于放入事件流非常重要,因为它允许下游会计服务将应付账款与收到的付款进行核对。
On the other hand, your business requirements may expect significant detail from the response of the request. A prime example of this is the use of a third-party payment service, where the input event states the amount that is due to be paid by the customer. The response payload from the third-party API is extremely important, as it specifies whether the payment succeeded or not, any error messages, and any additional details such as a unique, traceable number indicating the payment information. This data is important to put into an event stream, as it allows downstream accounting services to reconcile accounts payable with received payments.
分析事件可以捆绑在一起并定期批量发送,也可以在发生时发送。无论哪种情况,它们都会被发送到请求-响应 API,然后可以将它们路由到适当的事件流。如图 13-1所示,其中外部客户端应用程序将分析事件发送到事件接收器服务,事件接收器服务将它们路由到正确的输出事件流。
Analytical events may be bundled together and periodically sent in a batch or they may be sent as they occur. In either case, they will be sent to a request-response API, where they can then be routed on to the appropriate event streams. This is illustrated in Figure 13-1, where an external client application sends analytical events to an event receiver service that routes them to the correct output event stream.
在客户端生成事件时,使用模式对事件进行编码。这确保了高保真来源,减少下游消费者的误解,同时为生产者提供创建和填充事件的详细要求。
Use schemas to encode events when generating them on the client side. This ensures a high-fidelity source that reduces misinterpretation by downstream consumers, while giving producers detailed requirements for creating and populating their events.
模式化事件对于大规模使用分析事件至关重要。模式准确地阐明了收集的内容,以便用户可以在以后了解该事件。它们还提供了版本控制和演变的机制,并将填充、验证和测试事件的责任交给了数据的生产者(应用程序开发人员),而不是消费者(后端接收者和分析师)。确保事件在创建时遵循模式意味着接收者服务不再需要解释和解析事件,就像纯文本等格式的情况一样。
Schematized events are essential for consuming analytical events at scale. Schemas clarify exactly what is collected so users can make sense of the event at a later date. They also provide a mechanism for version control and evolution, and put the onus of populating, validating, and testing the event on the producer of that data (the application developers) and not the consumers (backend recipients and analysts). Ensuring that the event adheres to a schema at creation time means that the receiver service no longer needs to interpret and parse the event, as could be the case with a format such as plain text.
从运行多个版本代码的设备中提取分析事件时,必须考虑许多限制。例如,对于在最终用户的移动设备上运行的任何应用程序来说,这是一个特别常见的场景。添加新字段来收集新数据,或者停止收集其他事件数据当然是合理的。然而,虽然您可以通过锁定旧版本来强制用户升级其应用程序,但让他们针对每个小更改都更新其应用程序是不现实的。规划分析事件的多个版本,如图13-2所示。
There are a number of restrictions that you must account for when ingesting analytical events from devices running multiple versions of code. For instance, this is a particularly common scenario for any application running on an end user’s moble device. Adding new fields to collect new data, or ceasing the collection of other event data is certainly reasonable. However, while you could force users to upgrade their applications by locking out older versions, it’s not realistic to make them update their application for every small change. Plan for multiple versions of analytical events, as shown in Figure 13-2.
将外部事件源视为一组微服务实例。每个实例通过事件接收器服务将模式化事件生成到事件流中。
Think of external event sources as a set of microservice instances. Each instance produces schematized events into the event stream via the event receiver service.
最后,根据传入事件的模式和事件定义将传入事件分类到它们自己定义的事件流中非常重要。根据业务目的分离这些事件 ,就像分离任何其他 微服务的事件流一样。
Finally, it’s important to sort the incoming events into their own defined event streams based on their schemas and event definitions. Separate these events according to business purposes just as you would the event streams of any other microservice.
事件驱动的微服务通常需要通过请求-响应协议与第三方 API 进行通信。请求-响应模式非常适合事件驱动的处理;请求和响应被简单地视为远程函数调用。微服务基于事件驱动逻辑调用API并等待回复。收到回复后,微服务会解析它,确保它遵循预期的模式,并继续应用业务逻辑,就像它是任何其他事件一样。图 13-3显示了此过程的一般示例。
Event-driven microservices often need to communicate with third-party APIs via request-response protocols. The request-response pattern fits in nicely with event-driven processing; the request and response are treated simply as a remote function call. The microservice calls the API based on the event-driven logic and awaits the reply. Upon receipt of the reply, the microservice parses it, ensures it adheres to an expected schema, and continues applying business logic as though it were any other event. A generalized example of this process is shown in Figure 13-3.
以下源代码说明了使用阻塞调用的微服务的逻辑操作:
The following source code illustrates the logical operation of the microservice using a blocking call:
while(true){//endless processing loopEvent[]eventsToProcess=Consumer.consume("input-event-stream");for(Eventevent:eventsToProcess){// Apply business logic to current event to generate whatever// request is necessaryRequestrequest=generateRequest(event,...);// Make a request to the external endpoint. Indicate timeouts,// retries, etc.// This code uses a blocking call to wait for a response.Responseresponse=RequestService.makeBlockingRequest(request,timeout,retries,...);// HTTP response. If success, parse + apply business logic.if(response.code==200){// Parse the results into an object for use in this application<ClassType>parsedObj=parseResponseToObject(response);// Apply any more business logic if necessary.OutputEventoutEvent=applyBusinessLogic(parsedObj,event,...);// Write results to the output event stream.Producer.produce("output-stream-name",outEvent);}else{// Response is not 200.// You must decide how to handle these conditions.// Retry, fail, log, and skip, etc.}}// Commit the offsets only when you are satisfied with the processing results.consumer.commitOffsets();}
while(true){//endless processing loopEvent[]eventsToProcess=Consumer.consume("input-event-stream");for(Eventevent:eventsToProcess){// Apply business logic to current event to generate whatever// request is necessaryRequestrequest=generateRequest(event,...);// Make a request to the external endpoint. Indicate timeouts,// retries, etc.// This code uses a blocking call to wait for a response.Responseresponse=RequestService.makeBlockingRequest(request,timeout,retries,...);// HTTP response. If success, parse + apply business logic.if(response.code==200){// Parse the results into an object for use in this application<ClassType>parsedObj=parseResponseToObject(response);// Apply any more business logic if necessary.OutputEventoutEvent=applyBusinessLogic(parsedObj,event,...);// Write results to the output event stream.Producer.produce("output-stream-name",outEvent);}else{// Response is not 200.// You must decide how to handle these conditions.// Retry, fail, log, and skip, etc.}}// Commit the offsets only when you are satisfied with the processing results.consumer.commitOffsets();}
使用这种模式有很多好处。其一,它允许您在应用业务逻辑时将事件处理与请求响应 API 混合在一起。其次,您的服务可以根据需要调用任何它需要的外部 API。您还可以通过向端点发出许多非阻塞请求来并行处理事件。只有当每个请求都发送完毕后,服务才会等待结果;一旦获得它们,它就会更新偏移量并继续处理下一个事件批次。请注意,并行处理仅对队列式流有效,因为不保留处理顺序。
There are a number of benefits to using this pattern. For one, it allows you to mix event processing with request-response APIs while applying business logic. Second, your service can call whatever external APIs it needs, however it needs to. You can also process events in parallel by making many nonblocking requests to the endpoints. Only when each of the requests has been sent does the service wait for the results; once it obtains them, it updates the offsets and proceeds to the next event batch. Note that parallel processing is valid only for queue-style streams, since processing order is not preserved.
这种方法也有许多缺点。正如第 6 章中所讨论的,向外部服务发出请求将不确定性元素引入到工作流程中。重新处理事件,即使只是失败的批次,也可能会给出与原始处理期间进行的调用不同的结果。请务必在设计应用程序时考虑到这一点。如果请求-响应端点由组织外部的第三方控制,则对 API 或响应格式进行更改可能会导致微服务失败。
There are also a number of drawbacks to this approach. As discussed in Chapter 6, making requests to an external service introduces nondeterministic elements into the workflow. Reprocessing events, even just a failed batch, may give different results than the call made during original processing. Make sure to account for this when designing the application. In cases where the request-response endpoint is controlled by a third party external to your organization, making changes to the API or the response format can cause the microservice to fail.
最后,考虑向端点发出请求的频率。例如,假设您发现微服务中存在错误,并且需要倒带输入流以进行重新处理。事件驱动的微服务通常会按照执行代码的速度消耗和处理事件,这可能会导致发送到外部 API 的请求大幅增加。这可能会导致远程服务失败,或者可能会反应性地阻止来自您的 IP 地址的流量,从而导致微服务出现许多失败的请求和紧密的重试循环。您可以通过使用配额(请参阅“配额”)来限制消耗和处理速率来解决此问题,但这也需要处理请求的微服务进行严格限制。如果外部 API 超出组织的控制范围,则限制责任可能由您承担,并且可能需要在您的微服务中实现。当外部 API 能够提供大容量突发服务,但对超出基线的量收取不成比例的费用时(某些日志记录和指标服务可能会出现这种情况),这种情况尤其常见。
Finally, consider the frequency at which you make requests to an endpoint. For instance, say that you discover a bug in your microservice and need to rewind the input stream for reprocessing. Event-driven microservices typically consume and process events as fast as they can execute the code, which could lead to a massive surge in requests going to the external API. This can cause the remote service to fail or perhaps reactively block traffic coming from your IP addresses, resulting in many failed requests and tight retry loops by your microservice. You can somewhat address this issue by using quotas (see “Quotas”) to limit consumption and processing rates, but it will also require tight throttling by the microservice handling the requests. In the case of an external API outside of your organization’s control, the throttling responsibility may lie with you and may need to be implemented in your microservice. This is particularly common when the external API is capable of providing high-volume burst service, but charges you disproportionately for the volume exceeding the baseline, as can be the case with some logging and metric services.
您还可以使用本书到目前为止讨论的 EDM 原则创建事件驱动的微服务,为状态的随机访问提供请求-响应端点。微服务使用输入事件流中的事件,处理它们,应用任何业务逻辑,并根据应用程序需求在内部或外部存储状态。请求-响应 API 通常包含在应用程序中(本章稍后会详细介绍),它提供对这些底层状态存储的访问。这种方法可以分为两个主要部分:从内部状态存储提供状态,以及从外部状态存储提供状态。
You can also create event-driven microservices that provide a request-response endpoint for the random access of state by using the EDM principles discussed so far in this book. The microservice consumes events from input event streams, processes them, applies any business logic, and stores state either internally or externally according to application needs. The request-response API, which is often contained within the application (more on this later in the chapter), provides access to these underlying state stores. This approach can be broken down into two major sections: serving state from internal state stores, and serving state from external state stores.
微服务可以提供源自其内部状态的结果,如图13-4所示。客户端的请求被传递到负载均衡器,负载均衡器将请求路由到底层微服务实例之一。在这种情况下,只有一个微服务实例,并且由于它具体化了该应用程序的所有状态数据,因此其所有应用程序数据在该实例中都可用。此状态通过使用两个输入事件流(A 和 B)来实现,并将更改日志备份到事件代理。
Microservices can serve the results sourced from their internal state, as demonstrated in Figure 13-4. The client’s request is delivered to a load balancer that routes the request on to one of the underlying microservice instances. In this case, there is only one microservice instance, and since it is materializing all of the state data for this application, all of its application data is available within the instance. This state is materialized via the consumption of the two input event streams (A and B), with the changelog backed up to the event broker.
现在,需要多个微服务实例来处理负载并且内部状态可能在实例之间分割是很常见的。使用多个微服务实例时,您必须将状态请求路由到托管该数据的正确实例,因为所有内部状态都根据键进行分片,并且键值只能分配给一个分区。图 13-5显示了客户端发出请求,然后该请求被转发到包含必要状态的正确实例。
Now, it is quite common that multiple microservice instances are required to handle the load and that internal state may be split up between instances. When using multiple microservice instances, you must route requests for state to the correct instance hosting that data, as all internal state is sharded according to key, and a keyed value can only ever be assigned to one partition. Figure 13-5 shows a client making a request that is then forwarded to the correct instance containing the necessary state.
如果您的框架支持它们的使用,状态存储的热副本(请参阅“使用热副本”)也可以用于服务直接调用请求。请记住,热副本数据可能会与主状态存储的复制延迟成比例地过时。
Hot replicas of state stores (see “Using hot replicas”) may also be used to serve direct-call requests, should your framework support their use. Keep in mind that hot-replica data may be stale in proportion to the replication lag from the primary state store.
您可以依靠事件驱动处理的两个属性来确定哪个实例包含特定的键/值对:
There are two properties of event-driven processing that you can rely on to determine which instance contains a specific key/value pair:
一个键只能映射到一个分区(请参阅“重新分区事件流”)
A key can only be mapped to a single partition (see “Repartitioning Event Streams”)
一个分区只能分配给单个消费者实例(请参阅“作为事件流消费”)
A partition can only be assigned to a single consumer instance (see “Consuming as an event stream”)
消费者组内的微服务实例知道其分区分配及其对等体的分区分配。给定键的所有事件必须驻留在单个分区内,以便实现事件流。通过将分区器逻辑应用于请求的键,微服务可以生成该键的分区 ID 分配。然后,它可以交叉引用该分区 ID 与消费者组的分区分配, 以确定哪个实例包含与该键关联的具体化数据(如果该键的数据存在)。
A microservice instance within a consumer group knows its partition assignments and those of its peers. All events of a given key must reside inside a single partition for an event stream to be materialized. By applying the partitioner logic to the key of the request, the microservice can generate the partition ID assignment of that key. It can then cross-reference that partition ID with the partition assignments of the consumer group to determine which instance contains the materialized data associated with the key (if the data for that key exists at all).
例如,图 13-6说明了如何使用分区器分配的属性来路由 REST GET 请求。
For example, Figure 13-6 illustrates using the properties of the partitioner assignment to route a REST GET request.
分区器指示密钥在 P1 中,该 P1 被分配给实例 1。如果添加新实例并重新平衡分区,则后续路由可能需要转到不同的实例,这就是为什么消费者组分配非常有用确定分区分配的位置。
The partitioner indicates that the key is in P1, which is assigned to instance 1. If a new instance is added and the partitions are rebalanced, subsequent routing may need to go to a different instance, which is why the consumer group assignments are instrumental in determining the location of the partition assignments.
提供分片内部状态的一个缺点是,微服务实例数量越大,各个实例之间的状态就越分散。这减少了请求在第一次尝试时命中正确实例的几率,而无需重定向。如果负载均衡器只是在循环分配模式上运行并假设密钥均匀分布,则请求在第一次尝试时命中的机会可以表示为:
One drawback of serving sharded internal state is that the larger the microservice instance count, the more spread out the state between individual instances. This reduces the odds of a request hitting the correct instance on the first try, without needing a redirect. If the load balancer is simply operating on a round-robin distribution pattern and assuming an even distribution of keys, then the chance of a request being a hit on the first try can be expressed as:
事实上,对于大量实例,几乎所有请求都会导致未命中,然后进行重定向,从而增加响应的延迟和应用程序的负载(因为每个请求可能需要最多两次网络调用来处理它) ,而不是一个)。幸运的是,智能负载均衡器可以在向微服务发送初始请求之前执行路由逻辑,如图13-7所示。
In fact, for a very large number of instances, almost all requests will result in a miss followed by a redirect, increasing the latency of the response and load on the application (as each request will likely require up to two network calls to process it, instead of one). Luckily, a smart load balancer can perform the routing logic before sending the initial request to the microservices, as demonstrated in Figure 13-7.
智能负载均衡器应用分区器逻辑来获取分区 ID,将其与消费者组分配的内部表进行比较,然后相应地转发请求。分区分配需要从内部重新分区流或给定状态存储的变更日志流中推断出来。这种方法确实会将应用程序的逻辑与负载均衡器纠缠在一起,因此重命名状态存储或更改拓扑将导致转发失败。最好将任何智能负载均衡器纳入微服务的单一可部署和测试流程中,以便您可以在生产部署之前捕获这些错误。
The smart load balancer applies the partitioner logic to obtain the partition ID, compares it against its internal table of consumer group assignments, and then forwards the request accordingly. Partition assignments will need to be inferred from the internal repartition streams or the changelog streams for a given state store. This approach does entangle the logic of your application with the load balancer, such that renaming state stores or changing the topology will cause the forwarding to fail. It’s best if any smart load balancers are part of the single deployable and testing process of your microservice so that you can catch these errors prior to production deployment.
使用智能负载均衡器只是减少延迟的最佳方法。由于竞争条件和内部状态存储的动态重新平衡,每个微服务实例仍然必须能够重定向错误转发的请求。
Using a smart load balancer is just a best effort to reduce latency. Due to race conditions and dynamic rebalancing of internal state stores, each microservice instance must still be able to redirect incorrectly forwarded requests.
与内部状态存储方法相比,从外部状态存储提供服务有两个优点。其一,所有状态对每个实例都是可用的,这意味着请求不需要根据内部存储模型转发到托管数据的微服务实例。其次,消费者组重新平衡也不需要微服务在新实例中重新实现内部状态,因为所有状态都在实例外部维护。这使得微服务能够提供无缝扩展和零停机选项,而内部状态存储很难提供这些选项。
Serving from an external state store has two advantages over the internal state store approach. For one, all state is available to each instance, meaning that the request does not need to be forwarded to the microservice instance hosting the data as per the internal storage model. Second, consumer group rebalances also don’t require the microservice to rematerialize the internal state in the new instance, since again, all state is maintained external to the instance. This allows the microservice to provide seamless scaling and zero-downtime options that can be difficult to provide with internal state stores.
确保通过微服务的请求-响应 API 访问状态,而不是通过与状态存储的直接耦合。如果不这样做,就会引入共享数据存储,导致服务之间的紧密耦合,并使更改变得困难且存在风险。
Ensure that state is accessed via the request-response API of the microservice and not through a direct coupling with the state store. Failure to do so introduces a shared data store, resulting in tight coupling between services, and makes changes difficult and risky.
每个微服务实例使用并处理来自其输入事件流的事件,并将数据具体化到外部状态存储。每个实例还提供请求-响应 API,用于将具体化数据返回给请求客户端。这种模式(如图 13-8所示)反映了内部状态存储中的服务状态模式。请注意,每个微服务实例都可以为状态存储中的键控数据的整个域提供服务,因此可以处理传递给它的任何请求。
Each microservice instance consumes and processes events from its input event streams and materializes the data to the external state store. Each instance also provides the request-response API for serving the materialized data back to the requesting client. This pattern, shown in Figure 13-8, mirrors that of serving state from an internal state store. Note that each microservice instance can serve the entire domain of keyed data from the state store and thus can handle any request passed to it.
输入事件流处理和请求响应服务能力都通过增加或减少实例计数来扩展,就像内部状态存储微服务一样。实例的数量可以扩展到超出事件流分区的数量。这些额外的实例不会被分配要处理的分区,但它们仍然可以处理来自请求-响应 API 的外部请求。此外,它们作为备用实例存在,准备在其他实例之一丢失的情况下分配分区。
Both input event stream processing and request-response serving capacity scale by increasing or decreasing the instance count, as with internal state store microservices. The number of instances can be scaled beyond the count of the event stream partitions. These extra instances won’t be assigned partitions to process, but they can still process external requests from the request-response API. Furthermore, they exist as standby instances ready to be assigned a partition in the case that one of the other instances is lost.
这种模式的主要优点之一是它不需要太多的部署协调。这是一个单一的一体化微服务,无论当前实例数量如何,都可以继续从外部状态存储提供状态。
One of the main advantages of this pattern is that it doesn’t require much in the way of deployment coordination. This is a single all-in-one microservice that can continue to serve state from the external state store regardless of the current instance count.
在此模式中,请求-响应 API 与将状态具体化到外部状态存储的事件驱动微服务的可执行文件完全分开。请求-响应 API 仍然独立于事件处理器,尽管两者具有相同的有界上下文和部署模式。图 13-9 举例说明了这种模式。您可以看到如何通过单个 REST API 端点来处理请求,同时使用两个事件处理实例来处理事件。
In this pattern, the request-response API is completely separate from the executable of the event-driven microservice that materializes the state to the external state store. The request-response API remains independent from the event processor, though both have the same bounded context and deployment patterns. This pattern is exemplified in Figure 13-9. You can see how the requests are served via a single REST API endpoint, while events are processed using two event processing instances.
虽然此模式有两个微服务在单个数据存储上运行,但仍然只有一个有界上下文。这两个微服务被视为单个组合服务。它们驻留在同一代码存储库中,并一起测试、构建和部署。
While this pattern has two microservices operating on a single data store, there’s still just a single bounded context. These two microservices are treated as a single composite service. They reside within the same code repository and are tested, built, and deployed together.
该模型的主要优点之一是,由于请求-响应 API 完全独立于事件处理器,因此您可以独立选择实现语言和扩展需求。例如,您可以使用轻量级流框架来填充物化状态,但使用组织中已常用的语言和关联库来为客户提供一致的 Web 体验。这种方法可以为您带来两全其美的效果,尽管它确实会带来管理代码库中多个组件的额外开销。
One of the main advantages of this model is that because the request-response API is fully independent of the event processor, you can choose the implementation languages and scaling needs independently. For instance, you could use a lightweight stream framework to populate the materialized state, but use a language and associated libraries that are already commonly used in your organization to deliver a consistent web experience to your customers. This approach can give you the best of both worlds, though it does come with the additional overhead of managing multiple components in your codebase.
此模式的第二个主要优点是它将事件处理逻辑中的任何故障与请求响应处理应用程序隔离。这消除了事件处理代码中的任何错误或数据驱动问题可能导致请求响应处理实例崩溃的可能性,从而减少停机时间(请注意,状态将变得陈旧)。
A second major advantage of this pattern is that it isolates any failures in the event processing logic from the request-response handling application. This eliminates the chance that any bugs or data-driven issues in the event processing code could bring down the request-response handling instance, thereby reducing downtime (note that the state will become stale).
这种模式的主要缺点是复杂性和风险。协调两个独立应用程序之间的更改是有风险的,因为更改数据结构、拓扑和请求模式可能需要在两个服务中进行相关更改。此外,耦合服务使一些 EDM 原则失效,例如不通过公共数据存储共享状态以及对有界上下文使用单一可部署项。
The main disadvantages of this pattern are complexity and risk. Coordinating changes between two otherwise independent applications is risky, as altering the data structures, topologies, and request patterns can necessitate dependent changes in both services. Additionally, coupling services invalidates some of the EDM principles, such as not sharing state via common data stores and using singular deployables for a bounded context.
话虽如此,这仍然是实时提供数据的有用模式,并且经常在生产中成功使用。仔细管理部署和全面的集成测试是确保成功的关键。
All that being said, this is still a useful pattern for serving data in real time, and it is often successfully used in production. Careful management of deployments and comprehensive integration testing is key for ensuring success.
请求-响应 API 构成了许多系统之间通信的基础,因此,您需要确保您的应用程序能够以与事件驱动的微服务原则集成的方式处理这些数据输入。处理请求的一种方法就像处理任何非事件驱动系统一样:立即执行请求的操作并将响应返回给客户端。或者,您还可以将请求转换为事件,将其注入到自己的事件流中,然后像系统中的任何其他事件一样处理它。最后,微服务还可以执行这些操作的混合,通过仅将对业务重要的请求转换为事件(可以在有界上下文之外共享),同时同步处理其他请求。图 13-10说明了这个概念,稍后将在“示例:报纸出版工作流程(审批模式)”中对其进行扩展。
Request-response APIs form the basis of communications between many systems, and as a result, you need to ensure that your applications can handle these data inputs in a way that integrates with event-driven microservice principles. One way to handle requests is just as you would with any non-event-driven system: perform the requested operation immediately and return the response to the client. Alternately, you can also convert the request into an event, inject it into its own event stream, and process it just as any other event in the system. Finally, the microservice may also perform a mix of these operations, by turning only requests that are important to the business into events (that can be shared outside the bounded context), while handling other requests synchronously. Figure 13-10 illustrates this concept, which will be expanded on shortly in “Example: Newspaper publishing workflow (approval pattern)”.
上图最左侧部分显示了正在执行的传统对象创建操作,并将结果直接写入数据库。或者,最右边的部分显示了事件优先的解决方案,其中请求被解析为事件并发布到相应的事件流,然后事件驱动的工作流使用它、应用业务逻辑并将其存储在数据库中。
The leftmost portion of the preceding figure shows a traditional object creation operation being performed, with the results written directly to the database. Alternately, the rightmost portion shows an event-first solution, where the request is parsed into an event and published to a corresponding event stream, prior to the event-driven workflow consuming it, applying business logic, and storing it in the database.
首次写入事件流的主要好处是它提供了事件的持久记录,并允许任何服务具体化该数据。然而,最大的权衡是产生的延迟,并且服务必须等待结果具体化到要使用的数据存储中(写入后最终一致的读取)。缓解这种延迟的一种方法是在成功将值写入对象流后将其保留在内存中,以便您在应用程序端操作中使用它。然而,这不适用于需要数据存在于数据库中的操作(例如,joins),因为必须首先具体化事件。
The major benefit of first writing to the event stream is that it provides a durable record of the event, and allows any service to materialize off of that data. The biggest tradeoff, however, is the latency incurred, and that the service must wait for the result to be materialized into the data store to be used (eventually-consistent read-after-write). One way to mitigate this delay is to keep the value in memory after successfully writing it to the object stream, allowing you to use it in application-side operations. This will not, however, work for operations that require the data to be present in the database (e.g., joins), as the event must be materialized first.
用户界面 (UI) 是人们与服务的有界上下文交互的方式。请求-响应框架对于 UI 应用程序来说非常常见,有许多选项和语言可以满足用户的需求。将这些框架集成到事件驱动领域对于释放其内在价值非常重要。
A user interface (UI) is the means by which people interact with the bounded context of a service. Request-response frameworks are exceedingly common for a UI application, with many options and languages available to serve users’ needs. Integrating these frameworks into the event-driven domain is important for unlocking their intrinsic value.
将用户输入作为事件流处理时需要解决许多问题。将请求作为事件处理的应用程序设计必须包含异步 UI。您还必须确保应用程序行为管理用户的期望。例如,在同步系统中,单击按钮的用户可能期望在非常短的时间内(可能在 100 毫秒或更短时间内)收到失败或成功响应。在异步事件处理系统中,处理服务可能需要超过 100 毫秒的时间来处理和处理响应,特别是当事件流有大量记录需要处理时。
There are a number of concerns to address when processing user input as an event stream. Application designs that process requests as events must incorporate an asynchronous UI. You must also ensure that the application behavior manages user expectations. For example, in a synchronous system, a user that clicks a button may expect to receive a failure or success response in very short order, perhaps in 100 ms or less. In an asynchronous event processing system, it may take the processing service longer than 100 ms to process and handle the response, especially if the event stream has a large number of records to process.
将用户输入作为事件处理时,研究并实施异步 UI 的最佳实践。正确的 UI 设计可以让用户做好等待异步结果的准备。
Research and implement best practices for asynchronous UIs when handling user input as events. Proper UI design prepares the user to expect asynchronous results.
您可以使用某些异步 UI 技术来帮助管理用户的期望。例如,您可以更新 UI 以指示他们的请求已发送,同时阻止他们在请求完成之前执行任何其他操作。航空公司预订和汽车租赁网站通常会显示带有旋转轮符号的“请稍候”消息,从而将网页的其余部分从用户输入中删除。这通知用户后端服务正在处理该事件,并且在该事件完成之前他们无法执行任何其他操作。
There are certain asynchronous UI techniques you can use to help manage your users’ expectations. For example, you can update the UI to indicate that their request has been sent, while simultaneously discouraging them from performing any more actions until it has completed. Airline booking and automobile rental websites often display a “please wait” message with a spinning wheel symbol, blanking out the rest of the web page from user input. This informs users that the backend service is processing the event and that they can’t do anything else until it has completed.
另一个需要考虑的因素是微服务可能需要持续处理传入的非用户事件,同时等待进一步的用户输入。您必须决定服务的事件处理何时已充分进行,以便将更新推送到 UI。事实上,尽管大多数 EDM 服务必须处理持续的更新,但您还必须决定事件从一开始的初始处理何时能够跟上当前的情况。
Another factor to consider is that the microservice may need to continually process incoming nonuser events while awaiting further user input. You must decide when the service’s event processing has progressed sufficiently for an update to be pushed to the UI. In fact, you must also decide when the initial processing of events from the beginning of time has caught up to the present, despite the ongoing updates that most EDM services must handle.
没有硬性规则规定何时必须更新界面。有界上下文的业务规则当然可以为您提供指导,主要围绕用户根据当前状态做出决策的影响。回答以下问题可能会帮助您决定如何以及何时更新 UI:
There are no hard-and-fast rules dictating when you must update your interface. The business rules of the bounded context can certainly guide you, predominantly around the impact of users making decisions based on the current state. Answering the following questions may help you decide how and when to update your UI:
用户根据过时状态做出决策会产生什么影响?
What is the impact of the user making a decision based on stale state?
推送 UI 更新对性能/体验有何影响?
What is the performance/experience impact of pushing a UI update?
导致请求重试的间歇性网络故障可能会导致重复事件。确保您的消费者可以幂等地处理重复项,如“生成重复事件”中所述。
Intermittent network failures causing request retries can introduce duplicate events. Ensure that your consumers can handle duplicates idempotently, as covered in “Generating duplicate events”.
This next example demonstrates some of the benefits of converting requests directly to events prior to processing.
报纸出版商有一个应用程序来管理其出版物的布局。每个出版物都依赖可定制的模板来确定文章和广告的放置方式和位置。
A newspaper publisher has an application that manages the layout of its publications. Each publication relies upon customizable templates to determine how and where articles and advertisements are placed.
图形用户界面(GUI)允许报纸设计者根据出版商的业务逻辑来排列和放置文章。最热门的新闻放在头版,不太重要的文章放在后面。广告也根据自己的特定规则进行定位,通常取决于尺寸、内容、预算和投放协议。例如,一些广告商可能不希望他们的广告放置在特定类型的故事旁边(例如,儿童玩具公司希望避免将其广告放置在有关绑架的故事旁边)。
A graphical user interface (GUI) allows the newspaper designers to arrange and place articles according to the publisher’s business logic. The hottest news is placed on the front pages, with less important articles placed further in. Advertisements are also positioned according to their own specific rules, usually dependent on size, content, budget, and placement agreements. For example, some advertisers may not want their ads placed next to specific types of stories (e.g., a children’s toy company would want to avoid having its ad placed alongside a story about a kidnapping).
报纸设计师负责根据版面模板放置文章和广告。报纸编辑负责确保报纸具有凝聚力,文章按类别排序并预测对读者的重要性,并根据合同放置广告。报纸编辑必须批准设计师的作品才能发表,或者在需要重新组织的情况下拒绝该作品。图 13-11说明了此工作流程。
The newspaper designer is responsible for placing the articles and advertisements according to the layout template. The newspaper editor is responsible for ensuring that the newspaper is cohesive, that the articles are ordered by category and projected importance to the reader, and that the advertisements are placed according to the contracts. The newspaper editor must approve the work performed by the designer before it can be published, or reject the work in the case that a re-organization is required. Figure 13-11 illustrates this workflow.
编辑和广告商都可以拒绝拟议的报纸,尽管广告商只有在编辑已经批准布局的情况下才有机会这样做。此外,该报只对获得最重要的广告商的批准感兴趣,这些广告商的广告支出是重要的收入来源。
Both the editor and the advertiser can reject a proposed newspaper, though the advertiser will get the chance to do so only if the editor has already approved the layout. Furthermore, the newspaper is interested only in obtaining approval from the most important advertisers, those whose ad spend is a significant source of revenue.
报纸的设计和审批是两个独立的有界上下文,每个上下文都涉及自己的业务功能。这可以通过两个微服务进行镜像,如图13-12所示。为了简单起见,该图省略了帐户、帐户管理、身份验证和登录详细信息。
The design and the approval of the newspaper are two separate bounded contexts, each concerned with its own business functionality. This can be mirrored by two microservices, as shown in Figure 13-12. For simplicity’s sake, the figure omits accounts, account management, authentication, and login details.
此示例中有相当多的内容需要解压,因此让我们从报纸填充器微服务开始。该服务将布局模板、广告和文章流消耗到关系数据库中。在这里,负责布局的员工执行他们的任务,当报纸准备好审批时,他们将填充的报纸编译成 PDF,将其保存到外部存储,并将其生成到填充的报纸事件流。填充报纸事件的格式如下:
There is a fair bit to unpack in this example, so let’s start with the newspaper populator microservice. This service consumes layout templates, advertisements, and articles streams into a relational database. Here, the employee responsible for layout performs their tasks, and when the newspaper is ready for approval, they compile the populated newspaper into a PDF, save it to an external store, and produce it to the populated newspaper event stream. The format for the populated newspaper event is as follows:
//填充报纸事件
Key: String pn_key //填充的报纸键
价值: {
String pdf_uri //存储PDF的位置
int version //填充报纸的版本
Pages[] page_metadata //有关每个页面内容的元数据
- int 页码
- 枚举内容 //广告、文章
- String id //广告或文章的ID
}//Populated newspaper event
Key: String pn_key //Populated newspaper key
Value: {
String pdf_uri //Location of the stored PDF
int version //Version of the populated newspaper
Pages[] page_metadata //Metadata about what is on each page
- int page_number
- Enum content //Ad, article
- String id //ID of the ad or article
}
由于 PDF 可能太大而无法存储在事件中,因此可以将其存储在外部文件存储中,并通过通用资源标识符或 URI 提供访问权限(请参阅“最小化事件的大小” )。
Because the PDF may be too large to store in an event, it can be stored in an external file store, with access provided via a universal resource identifier, or URI (see “Minimize the Size of Events”).
您可能已经注意到,这个微服务没有将报纸填充器 GUI 的人机交互转换为事件 - 这是为什么?尽管“人类交互作为事件”是该示例的主题之一,但没有必要将所有人类交互转换为事件。这个特定的有界上下文实际上只涉及生成最终填充的报纸事件,但它是如何产生的并不是特别重要。这种责任封装允许您利用具有同步 GUI 模式的整体框架来构建此微服务,并使用您或您的开发人员可能已经熟悉的模式和软件技术。
You may have noticed that this microservice does not translate the human interactions of the newspaper populator GUI into events—why is this? Despite “human interactions as events” being one of the main themes of this example, it is not necessary to convert all human interaction into events. This particular bounded context is really only concerned with producing the final populated newspaper event, but it isn’t particularly important how it came to be. This encapsulation of responsibility allows you to leverage a monolithic framework with synchronous GUI patterns for building this microservice, and to use patterns and software technologies that you or your developers may already be familiar with.
填充的报纸流可能与报纸填充器微服务内的状态不同步。有关从整体进行原子生产的详细信息,特别是使用发件箱表模式或更改数据捕获日志,请参阅“数据解放模式” 。
The populated newspaper stream might get out of sync with the state within the newspaper populator microservice. See “Data Liberation Patterns” for details on atomic production from a monolith, particularly using the outbox table pattern or change-data capture logs.
批准由单独的微服务处理,其中填充的报纸事件由编辑加载以进行查看和批准。编辑者可以根据需要标记 PDF 副本、添加注释并提供初步批准以将其移至下一步:广告商批准。编辑者还可以在工作流程的任何阶段(在获得广告商审核之前、期间或之后)拒绝它。事件结构如下:
Approvals are handled by a separate microservice, where the populated newspaper event is loaded by the editor to view and approve. The editor can mark up a copy of the PDF as necessary, add comments, and provide tentative approval to move it on to the next step: advertiser approval. The editor may also reject it at any point of the workflow, before, during, or after obtaining advertiser review. The event structure is as follows:
//编辑批准事件
Key: String pn_key //填充的报纸键
价值: {
String Marked_up_pdf_uri //标记PDF的可选URI
int version //填充报纸的版本
枚举状态 //awaiting_approval、已批准、已拒绝
字符串 editor_id
字符串 editor_comments
RejectedAdvertisements[]rejectedAds //可选,如果被拒绝
- int 页码
- 字符串advertisement_id
- 字符串advertiser_id
- 字符串advertiser_comments
}//Editor approval event
Key: String pn_key //Populated newspaper key
Value: {
String marked_up_pdf_uri //Optional URI of the marked-up PDF
int version //Version of the populated newspaper
Enum status //awaiting_approval, approved, rejected
String editor_id
String editor_comments
RejectedAdvertisements[] rejectedAds //Optional, if rejected
- int page_number
- String advertisement_id
- String advertiser_id
- String advertiser_comments
}
向广告商提供用于批准其广告尺寸和位置的用户界面。该服务负责确定哪些广告需要批准,哪些不需要,并将 PDF 分割成适当的部分以供广告商查看。重要的是不要泄露有关新闻报道或竞争对手广告的信息。批准事件写入广告商的批准流,类似于编辑器的批准流:
Advertisers are provided with a UI for approving their advertisement size and placement. This service is responsible for determining which advertisements require approval and which do not, and for cutting up the PDF into appropriate pieces for the advertiser to view. It is important to not leak information about news stories or competitors’ advertisements. Approval events are written to an advertiser’s approval stream, similar to that of the editor:
//广告商批准事件
Key: String pn_key //填充的报纸键
价值: {
String adverter_pdf_uri //向广告商展示的PDF片段
int version //填充报纸的版本
int 页码
booleanapproved //批准或不批准
字符串advertisement_id
String Advertisingr_id //审批者ID
字符串advertiser_comments
}//Advertiser approval event
Key: String pn_key //Populated newspaper key
Value: {
String advertiser_pdf_uri //The PDF piece shown to the advertiser
int version //Version of the populated newspaper
int page_number
boolean approved //Approved or not
String advertisement_id
String advertiser_id //ID of the approver
String advertiser_comments
}
您可能已经注意到,广告商批准是关键的pn_key,并且每个报纸都会有多个广告商事件具有相同的关键。在这种情况下,广告商的批准被视为事件而不是实体,并且正是这些事件的聚合决定了广告商对报纸的完全批准。请记住,每个广告商都会登录其 GUI 并分别批准其广告,直到他们全部回复(或者可能未能及时回复)后,该流程才能进入最终批准阶段。如果您查看编辑器批准事件定义,您可以看到拒绝事件的聚合表示为RejectedAdvertisements对象数组。
You may have noticed that the advertiser approvals are keyed on pn_key and that there will be multiple advertiser events with this same key per newspaper. In this case the advertiser approvals are being treated as events and not entities, and it is the aggregate of these events that determines the complete approval by an advertiser for the newspaper. Keep in mind that each advertiser logs into their GUI and approves their ads separately, and it’s not until they have all replied (or perhaps, failed to reply in time) that the process can move on to the final approval. If you take a look at the editor approval event definition, you can see that the aggregation of rejected events is represented as an array of RejectedAdvertisements objects.
将报纸、编辑认可和广告商认可作为事件进行填充的好处之一是,它们共同构成了报纸、拒绝、评论和认可的规范叙述。您可以随时审核此叙述,以查看提交和批准的历史记录,并查明可能出现问题的位置。另一个好处是,通过直接写入事件,审批微服务可以使用纯流处理库(例如 Apache Kafka 或 Samza),在应用程序启动时直接从事件流具体化状态。无需创建外部状态存储来管理此数据。
One benefit of having populated newspaper, editor approval, and advertiser approval as events is that together they form the canonical narrative of newspapers, rejections, comments, and approvals. You can audit this narrative at any point in time to see the history of submissions and approvals, and pinpoint where things may have gone wrong. Another benefit is that by writing directly to events, the approval microservice can use a pure stream processing library, like Apache Kafka or Samza, to materialize the state directly from the event stream whenever the application starts up. There is no need to create an external state store for managing this data.
业务需求要求编辑审批服务和广告商审批服务分开。其中每一个都服务于一个相关但独立的业务环境。具体而言,当前组合服务的广告商组件负责:
Business requirements demand that the editor approval service and advertiser approval service be separated. Each of these serves a related, though separate, business context. In particular, the advertiser components of the currently combined service are responsible for:
确定要请求批准的广告商
Determining which advertisers to ask for approval
将 PDF 分割成可查看的块
Slicing up the PDF into viewable chunks
管理面向广告商的组件、控件和品牌
Managing advertiser-facing components, controls, and branding
处理面向公众的更广泛的互联网暴露,特别是在安全实践和软件补丁方面
Handling public-facing exposure to the wider internet, particularly around security practices and software patches
另一方面,组合服务的编辑器组件不需要解决面向公众的问题,例如形象、品牌和安全。它主要涉及:
The editor components of the combined service, on the other hand, do not need to address public-facing concerns such as image, branding, and security. It is primarily concerned with:
批准总体布局、设计和流程
Approving overall layout, design, and flow
评估零售商回复的摘要(不是单独评估每个零售商的回复)
Assessing the summary of retailer responses (not each one individually)
向报纸设计者提供如何适应广告商拒绝的建议
Providing suggestions to the newspaper designer on how to accommodate advertiser rejections
新微服务布局的模型如图13-13所示。
A mock-up of the new microservice layout is shown in Figure 13-13.
有两个新的事件流需要考虑。第一个是在步骤 2 中,编辑器批准的 pn 流。该流的格式与填充的报纸流的格式相同,但仅在编辑对整个报纸感到满意并将其发布以供广告商批准后才会生成该事件。
There are two new event streams to consider. The first is in step 2, the editor-approved p.n. stream. The format of this stream is identical to that of the populated newspaper stream, but this event is produced only after the editor is satisfied with the overall newspaper and releases it for advertiser approval.
填充的报纸流是所有候选报纸的唯一事实来源。编辑批准的 pn 流是仅适用于已批准广告商审查的报纸的单一事实来源,已被编辑系统的逻辑过滤。这两个事件流没有相同的业务 含义。
The populated newspaper stream is the single source of truth for all candidate newspapers. The editor-approved p.n. stream is the single source of truth only for newspapers that have been approved for advertiser review, having been filtered by the editor system’s logic. The two event streams do not have the same business meaning.
这种设计的一个主要优点是所有编辑者门控逻辑完全保留在编辑者审批服务内。请注意,对填充的报纸流的更新不会自动转发,而是依赖于编辑发布它们以获得批准。同一份报纸 ( pn_key) 的多个版本完全包含在编辑服务中。这种安排让编辑可以控制发送哪些版本以供批准,同时限制任何进一步的修订,直到他们对最初的广告商反馈感到满意为止。
A major advantage of this design is that all editor gating logic stays completely within the editor approval service. Note that updates to the populated newspaper stream are not automatically forwarded on, but rely on the editor releasing them for approval. Multiple versions of the same newspaper (pn_key) are contained entirely within the editor service. This arrangement lets the editor control which versions are sent on for approval, while gating any further revisions until they are satisfied with the initial advertiser feedback.
第二个新事件流位于步骤 3 中,即广告批准摘要流。它包含广告商批准服务的结果摘要,旨在提供给定报纸的每个响应的历史记录和当前状态。请记住,作为一项单独的服务,编辑批准服务无法知道哪些广告商已收到批准其广告的指令。该信息严格属于广告审批系统的范围,尽管它可以向编辑传达结果摘要。广告审批摘要事件的格式如下:
The second new event stream is in step 3, the ad-approvals summary stream. It contains the summaries of the results from the advertiser approval service, designed to provide both a historical record and the current status of each of the responses for a given newspaper. Keep in mind that as a separate service, the editor approval service has no way to know which advertisers have been sent instructions to approve their advertisements. That information is strictly the domain of the advertising approval system, though it can communicate a summary of the results to the editor. The format of the ad-approval summary event is as follows:
//广告审批摘要事件
键:字符串 pn_key
价值: {
int version //填充报纸的版本
AdApprovalStatus[] ad_app_status
- 枚举状态//等待、已批准、已拒绝、超时
- int 页码
- 字符串advertisement_id
- 字符串advertiser_id
- 字符串advertiser_comments
}//Ad-approval summary event
Key: String pn_key
Value: {
int version //Version of the populated newspaper
AdApprovalStatus[] ad_app_status
- Enum status //Waiting, Approved, Rejected, Timedout
- int page_number
- String advertisement_id
- String advertiser_id
- String advertiser_comments
}
该广告批准摘要事件定义演示了将广告商批准状态封装到广告商批准服务中。编辑可以根据广告审批摘要事件的状态来决定报纸的审批,而无需管理或处理获得这些结果的任何工作。
This ad-approval summary event definition demonstrates the encapsulation of advertiser approval state into the advertiser approval service. The editor can make decisions on the approval of the newspaper based on the statuses of the ad-approval summary event, without having to manage or handle any of the work of obtaining those results.
前端和后端服务通过三种主要方式进行协调,为用户带来业务价值。单体后端在许多任何规模的组织中都很常见。随着微服务(同步和事件驱动)的日益普及,微服务后端变得越来越流行。在前两种方法中,前端和后端服务由不同的团队拥有和运营,因此端到端业务功能跨越团队边界。相比之下,微前端方法从后端到前端完全根据业务问题调整实现。这三种方法如图 13-14所示。
Frontend and backend services coordinate in three primary ways to bring business value to users. Monolithic backends are common in many organizations of any size. Microservice backends have become more popular with the growing adoption of microservices, both synchronous and event-driven. In both of these first two approaches, the frontend and backend services are owned and operated by separate teams, such that the end-to-end business functionality crosses team boundaries. In contrast, a microfrontends approach aligns implementations completely on business concerns, from backend to frontend. These three approaches are illustrated in Figure 13-14.
整体后端方法是大多数软件开发人员所熟悉的一种方法,至少在某种程度上是这样。在许多情况下,一个专门的后端团队(通常由许多针对非常大的单体的子团队组成)执行单体上的大部分工作。随着整体规模的增长,员工数量也会增加。
The monolithic backend approach is one that most software developers are familiar with, at least to some extent. In many cases, a dedicated backend team, usually composed of many subteams for very large monoliths, performs most of the work on the monolith. Head count increases as the monolith grows.
前端团队与后端完全分离,他们通过请求-响应 API 进行通信,以获取呈现客户 UI 所需的数据。在此架构中实现的产品必须协调团队之间和跨技术实现的工作,这使其可能成为最昂贵的功能交付方式之一。
The frontend team is completely separate from the backend, and they communicate via a request-response API to obtain the necessary data for rendering the customer’s UI. A product implemented in this architecture has to coordinate efforts between teams and across technical implementations, making it potentially one of the most expensive ways of delivering functionality.
微服务后端方法是许多团队迁移到微服务的最终选择,无论好坏,他们中的许多人都会选择这种方法。这种方法的主要优点是后端现在由独立的、以产品为中心的微服务组成,每个微服务(或一组支持产品的微服务)由单个团队独立拥有。每个微服务都会具体化必要的数据,执行其业务逻辑,并向聚合层公开任何必要的请求-响应 API 和事件流。
The microservice backend approach is one where many teams migrating to microservices eventually end up, and for better or worse, it is where many of them stay. The major advantage of this approach is that the backend is now composed of independent, product-focused microservices, with each microservice (or set of product-supporting microservices) independently owned by a single team. Each microservice materializes the necessary data, performs its business logic, and exposes any necessary request-response APIs and event streams up to the aggregation layer.
微服务后端方法的一个主要缺点是它仍然严重依赖聚合层,而聚合层可能会出现许多问题。由于尝试解决产品边界问题或通过合并其他独立产品的功能来实现“快速获胜”,业务逻辑可能会渗透到这一层。这一层经常遭受公地悲剧,即每个人都依赖它,但没有人对此负责。虽然这可以在一定程度上通过严格的管理模型来解决,但看似无害的微小变化的积累仍然可能导致不适当的业务逻辑泄漏。
A major downside of the microservice backend approach is that it still depends heavily on an aggregation layer, where numerous problems can pop up. Business logic can creep into this layer due to attempts to resolve product boundary issues, or to achieve “quick wins” by merging features of otherwise separate products. This layer often suffers from the tragedy of the commons, whereby everyone relies on it but no one is responsible for it. While this can be resolved to some extent by a strict stewardship model, accumulations of minor, seemingly innocent changes can still let an inappropriate amount of business logic leak through.
第三种方法是微前端,它将整体前端拆分为一系列独立的组件,每个组件都由后端微服务提供支持。
The third approach, the microfrontend, splits up the monolithic frontend into a series of independent components, each backed by supporting backend microservices.
微前端模式与事件驱动的微服务后端非常匹配,并继承了它们的许多优点,例如模块化;业务关注点分离;自治团队;以及部署、语言和代码库的独立性。
Microfrontend patterns match up very well with event-driven microservice backends, and inherit many of their advantages, such as modularity; separation of business concerns; autonomous teams; and deployment, language, and code-base independence.
让我们看看微前端的其他一些显着好处。
Let’s look at some of the other notable benefits of microfrontends.
微前端是一种组合模式,这意味着您可以根据需要向现有 UI 添加服务。值得注意的是,微前端与事件驱动的后端配合得非常好,事件驱动的后端本质上也是基于组合的。事件流使微服务能够提取支持后端有界上下文所需的事件和实体。后端服务可以专门针对微前端提供的产品的业务需求构建必要的状态并应用业务逻辑。可以选择状态存储实现来专门满足服务的要求。这种组合形式为如何构建前端服务提供了巨大的灵活性,正如您将在“示例:体验搜索和评论应用程序”中看到的那样。
Microfrontends are a compositional pattern, meaning you can add services as needed to an existing UI. Notably, microfrontends pair extremely well with event-driven backends, which are also intrinsically composition-based. Event streams enable the microservice to pull in the events and entities needed to support the backend bounded context. The backend service can construct the necessary state and apply business logic specifically for the business needs of the product provided by the microfrontend. The state store implementation can be selected to specifically suit the requirements of the service. This form of composition provides tremendous flexibility in how frontend services can be built, as you’ll see in “Example: Experience Search and Review Application”.
通过严格按照业务边界上下文调整微前端,就像处理在后端运行的其他微服务一样,您可以直接跟踪特定的业务需求及其实现。这样,您可以轻松地将实验产品注入到应用程序中,而不会对 现有核心服务的代码库产生不利影响。如果它们的性能或用户使用率不符合预期,您可以轻松删除它们。这种协调和隔离确保了来自不同工作流程的产品需求不会相互渗透。
By aligning microfrontends strictly on business bounded contexts, just as you’d do with other microservices operating in the backend, you can trace specific business requirements directly to their implementations. This way, you can easily inject experimental products into an application without adversely affecting the codebase of existing core services. And should their performance or user uptake not be as expected, you can just as easily remove them. This alignment and isolation ensures that product requirements from various workflows do not bleed into one another.
虽然微前端可以分离业务问题,但您必须考虑在整体前端中可能认为理所当然的功能,例如一致的 UI 元素和对每个元素布局的完全控制。微前端还继承了所有微服务共有的一些问题,例如潜在的重复代码以及管理和部署微服务的操作问题。本节涵盖了一些特定于微前端的注意事项。
While microfrontends enable separation of business concerns, you have to account for features that you may take for granted in a monolithic frontend, such as consistent UI elements and total control over each element’s layout. Microfrontends also inherit some of the issues common to all microservices, such as potential for duplicated code and the operational concerns of managing and deploying microservices. This section covers a few microfrontend-specific considerations.
应用程序的视觉风格保持一致非常重要,当前端体验由许多独立的微前端组成时,这可能具有挑战性。每个微前端都是另一个潜在的故障点,即 UI 设计可能与所需的用户体验不一致。解决这个问题的一种方法是提供强大的样式指南,以及每个微前端中使用的常用 UI 元素的精简库。
It’s important that the applications’ visual style remains consistent, and this can be challenging when a frontend experience is composed of many independent microfrontends. Each microfrontend is another potential point of failure—that is, where the UI design might be inconsistent with the desired user experience. One method to remedy this is to provide a strong style guide, in conjunction with a lean library of common UI elements to be used in each microfrontend.
这种方法的缺点是它需要密切维护样式指南和元素的所有权。使用产品中的元素库在多个团队之间协调添加新元素和修改现有元素可能有点困难。使用管理模型(类似于许多流行的开源项目中使用的模型)来容纳这些资产,可以帮助确保以经过衡量和深思熟虑的方式完成更改。这需要资产用户之间的参与和对话,因此会产生管理成本。
The downside to this approach is that it requires closely maintaining ownership of both the style guide and the elements. Adding new elements and modifying existing ones can be a bit difficult to coordinate across multiple teams using the element library in their products. Accommodating these assets using a stewardship model, similar to that used in many popular open source projects, can help ensure that changes are done in a measured and deliberate way. This requires participation and dialogue between the asset users and, as a result, incurs an overhead cost.
确保通用 UI 元素库不包含任何特定于有界上下文的业务逻辑。将所有业务逻辑封装在其自己适当的有界上下文中。
Ensure common UI element libraries are free of any bounded-context-specific business logic. Keep all business logic encapsulated within its own proper bounded context.
最后,对应用程序的通用 UI 元素进行更改可能需要重新编译和重新部署每个微前端。这在运营上可能会很昂贵,因为每个微前端团队都需要更新其应用程序,进行测试以确保 UI 符合新要求,并验证它是否按预期与将其缝合在一起的 UI 层集成(接下来将详细介绍) 。由于很少进行全面的 UI 更改,因此在一定程度上减轻了这一费用。
Finally, making changes to the common UI elements of the application may require that each microfrontend be recompiled and redeployed. This can be operationally expensive, as each microfrontend team will need to update its application, test to ensure that the UI adheres to the new requirements, and verify that it integrates as expected with the UI layer that stitches it together (more on this next). This expense is somewhat mitigated by the infrequency of sweeping UI changes.
微前端作为复合框架的一部分,有时可能会出现问题。这些单独的前端可能以不同的速率加载,或更糟糕的是,在故障期间可能根本不加载任何内容。您必须确保复合前端可以优雅地处理这些场景,并且仍然为仍在工作的部分提供一致的体验。例如,您可能希望对仍在等待慢速微前端结果的元素使用旋转的“正在加载”标志。将这些微前端拼接在一起是正确 UI 设计的一个练习,但这个过程的更深层次的细节和细微差别超出了本书的范围。
Microfrontends, as pieces of a composite framework, can be problematic at times. These separate frontends may load at different rates, or worse, may not load anything at all during a failure. You must ensure that the composite frontend can handle these scenarios gracefully and still provide a consistent experience for the parts of it that are still working. For example, you may want to use spinning “loading” signs for elements that are still awaiting results from slow microfrontends. Stitching these microfrontends together is an exercise in proper UI design, but the deeper details and nuances of this process are beyond the scope of this book.
“一次经历是你永远不会忘记的!” 该应用程序的开发者称,该应用程序将度假者与当地指南、景点、娱乐和美食联系起来。用户可以搜索当地体验,获取详细信息和联系信息,并留下评论。
“An experience is something you’ll never forget!” claim the makers of the application, which connects vacationers with local guides, attractions, entertainment, and culinary delights. Users can search for local experiences, obtain details and contact information, and leave reviews.
该应用程序的第一个版本具有单个服务,可将体验实体和客户评论具体化到单个端点中。用户可以输入他们的城市名称来查看他们所在地区的可用体验列表。一旦他们选择了一个选项,就会显示体验信息以及任何相关的评论,如图13-15中的简单模型所示。
The first version of this application has a single service that materializes both the experience entities and customer reviews into a single endpoint. Users can input their city name to see a list of available experiences in their area. Once they select an option, the experience information along with any associated reviews are displayed, as in the simple mockup in Figure 13-15.
在应用程序的第一个版本中,数据存储在基本键/值状态存储中,仅提供有限的搜索功能。基于用户地理位置的搜索尚不可用,尽管这是您的用户一直在请求的功能。此外,对于版本 2 来说,将评论拆分到自己的微服务中是一个好主意,因为它们具有足够不同的业务职责来形成自己的有界上下文。最后,您应该创建产品微前端来将这两个产品缝合在一起,并充当每个业务服务的聚合层。这三个微前端中的每一个都可能由自己的团队或同一团队拥有和管理,尽管关注点分离允许像后端微服务一样扩展所有权。图 13-16显示了一个新的 GUI 模型,显示了分离的前端职责。
In the first version of the application, data is stored in a basic key/value state store that offers only limited searching capabilities. Searching based on the user’s geolocation is not yet available, though it is something your users have been requesting. Additionally, it would be a good idea for version 2 to split off reviews into their own microservice, as they have sufficiently distinct business responsibilities to form their own bounded context. Finally, you should create the product microfrontend to stitch these two products together and act as the aggregation layer for each business service. Each of these three microfrontends may be owned and managed by their own team, or the same team, though the separation of concerns allows for scaling ownership just as in backend microservices. A new mockup of the GUI showing the separated frontend responsibilities is shown in Figure 13-16.
现在,产品边界封装了搜索和评论微前端,并包含将这两种服务缝合在一起所需的所有逻辑。但是,它不包含与这些服务相关的任何业务逻辑。此更新的 UI 还说明了微前端的职责如何变化,因为它现在必须支持地理位置搜索功能。用户的地址被转换为经纬度坐标,可用于计算到附近体验的距离。同时,评论微前端的职责保持不变,但它摆脱了与搜索服务的耦合。图 13-17显示了向微前端的迁移的外观。
Now the product boundary encapsulates both the search and review microfrontends and contains all the logic necessary to stitch these two services together. It does not, however, contain any business logic pertaining to these services. This updated UI also illustrates how the microfrontend’s responsibilities have changed, as it must now support geolocation search functionality. The user’s address is transposed into lat-lon coordinates, which can be used to compute the distance to nearby experiences. Meanwhile, the review microfrontend’s responsibilities remain the same, but it is freed of its coupling to the search service. Figure 13-17 shows how this migration into microfrontends could look.
这个数字有几个值得注意的点。首先,正如本章前面所讨论的,评论首先作为事件发布到评论事件流,然后再摄取回数据存储中。这对于服务的两个版本都是如此,它说明了将核心业务数据保留在实施之外的重要性。通过这种方式,您可以轻松地将审阅服务分解为自己的微服务,而无需通过数据同步执行不必要且容易出错的操作。
There are a few notable points about this figure. First, as discussed earlier in this chapter, the reviews are being published first as events to the review event stream, and then subsequently ingested back into the data store. This is true for both versions of the service, and it illustrates the importance of keeping core business data external to the implementation. In this way you can easily break out the review service into its own microservice, without performing unnecessary and error-prone operations with data synchronization.
如果评论保留在版本 1 的数据存储内部,那么您将不得不考虑将它们释放以供版本 2 使用(第 4 章),然后为其在事件流中的长期存储制定迁移计划。
If the reviews were kept internal to version 1’s data store, you would instead have to look into liberating them for version 2’s use (Chapter 4) and then come up with a migration plan for its long-term storage in an event stream.
无论服务需要什么,实现和使用任何业务事件流的能力使得 事件驱动的微服务后端与 微前端如此有效地配对。
The ability to materialize and consume any stream of business events, however the service needs them, is what makes event-driven microservice backends pair so effectively with microfrontends.
其次,评论服务已分解为自己的微服务,将其有界上下文和实现与搜索完全分离。第三,搜索服务已将其状态存储替换为同时具有纯文本和 地理位置搜索功能的存储。此更改支持搜索服务的业务需求,现在可以独立于审阅服务业务需求来解决搜索服务的业务需求。该解决方案说明了基于组合的后端如何使开发团队能够灵活地使用最佳工具来支持微前端产品。
Second, the review service has been broken out into its own microservice, fully separating its bounded context and implementation from those of search. Third, the search service has replaced its state store with one capable of both plain-text and geolocation search functionality. This change supports the business requirements of the search service, which can now be addressed independently of the review service business requirements. This solution illustrates how composition-based backends give development teams the flexibility to use the best tools to support the microfrontend product.
在此新版本中,搜索微服务使用用户配置文件实体流中的事件来个性化搜索结果。虽然后端服务的版本 1 当然也可以消费和使用这些数据,但版本 2 中增加的服务粒度明确了哪些业务功能正在使用用户数据。观察者只需查看有界上下文的输入流,就可以知道前端的每个部分消耗和使用了哪些流。相反,在版本 1 中,如果不深入研究代码,观察者将不知道是使用用户事件的搜索部分还是审阅部分。
In this new version, the search microservice consumes events from the user profile entity stream to personalize search results. While version 1 of the backend service could certainly also consume and use this data, the increased granularity of the services in version 2 clarifies which business functions are using the user data. An observer can tell which streams are consumed and used for each part of the frontend just by looking at the input streams for the bounded contexts. Conversely, in version 1, without digging into the code, an observer would have no idea whether it’s the search or the review portion using the user events.
最后,请注意,旧版本和新版本的所有必要数据都源自完全相同的事件流。由于这些事件流是单一事实来源,因此您可以更改应用程序后端,而不必担心维护特定状态存储实现或迁移数据。这与整体后端形成鲜明对比,在整体后端中,数据库还扮演着数据通信层的角色,并且不能轻易更换。事件驱动后端与微前端的组合仅受可用事件数据的粒度和细节的限制。
Finally, note that all necessary data for both the old and new versions are sourced from the exact same event streams. Because these event streams are the single source of truth you can change the application backends without having to worry about maintaining a specific state store implementation or about migrating data. This is in stark contrast to a monolithic backend, where the database also plays the role of data communication layer and cannot be easily swapped out. The combination of an event-driven backend paired with a microfrontend is limited only by the granularity and detail of the available event data.
本章介绍了事件驱动的微服务与请求-响应 API 的集成。外部系统主要通过请求-响应 API 进行通信,无论是人驱动还是机器驱动,它们的请求和响应可能必须转换为事件。机器输入可以提前进行模式化,以发出可以通过请求响应 API 在服务器端收集的事件。第三方 API 通常需要解析响应并将其包装到自己的事件定义中,并且往往会因更改而变得更加脆弱。
This chapter has covered the integration of event-driven microservices with request-response APIs. External systems predominantly communicate via request-response APIs, be they human or machine driven, and their requests and responses may have to be converted into events. Machine input can be schematized ahead of time, to emit events that can be collected server-side via the request-response API. Third-party APIs typically require parsing and wrapping the responses into their own event definition and tend to be more brittle with change.
人类交互也可以转换为事件,由消费事件驱动的微服务异步处理。这需要集成设计,其中用户界面提示用户他们的请求正在异步处理。通过将所有重要的用户输入视为事件流,有界上下文的实现可以有效地与用户数据解耦。这使得设计的架构演进具有显着的灵活性,并允许在没有过度困难的情况下更改组件。
Human interactions can also be converted into events, to be processed asynchronously by the consuming event-driven microservice. This requires an integrated design, where the user interface cues the user that their request is being handled asynchronously. By treating all essential user inputs as streams of events, the implementation of the bounded context is effectively decoupled from the user data. This allows for significant flexibility in the architectural evolution of the design and permits components to be altered without undue hardship.
最后,微前端为基于事件驱动的微服务的产品全栈开发提供了架构。后端事件驱动的微服务本质上是组合的,将事件和实体聚集在一起以应用业务逻辑。这种模式扩展到前端,其中用户体验不需要是一个大型的整体应用程序,而是可以损害许多专门构建的微前端。每个微前端都服务于其特定的业务逻辑和功能,并通过整体组合层将各种应用程序缝合在一起。这种架构风格反映了后端微服务的自主性和部署模式,提供了完整的产品一致性,并允许灵活的前端选项用于实验、细分和交付自定义用户体验。
Finally, microfrontends provide an architecture for full-stack development of products based on event-driven microservices. Backend event-driven microservices are compositional by nature, drawing together events and entities to apply business logic. This pattern is extended to the frontend, where user experiences need not be one large monolithic application, but instead can compromise a number of purpose-built microfrontends. Each microfrontend serves its particular business logic and functionality, with an overall compositional layer to stitch the various applications together. This architectural style mirrors the autonomy and deployment patterns of the backend microservices, providing full product alignment and allowing flexible frontend options for experimentation, segmentation, and delivery of custom user experiences.
支持工具使您能够有效地大规模管理事件驱动的微服务。虽然其中许多工具可以通过管理员执行的命令行界面提供,但最好拥有一系列自助服务工具。这些提供了对于确保可扩展和弹性业务结构至关重要的 DevOps 功能。本章介绍的工具绝不是唯一可用的工具,但我和其他人根据我们的经验发现它们是有用的。您的组织需要决定为自己的用例采用什么。
Supportive tooling enables you to efficiently manage event-driven microservices at scale. While many of these tools can be provided by command-line interfaces executed by administrators, it is best to have a gamut of self-serve tools. These provide the DevOps capabilities that are essential for ensuring a scalable and elastic business structure. The tools covered in this chapter are by no means the only ones available, but they are tools that I and others have found useful in our experience. Your organization will need to decide what to adopt for its own use cases.
不幸的是,缺乏用于管理事件驱动微服务的免费开源工具。在适用的情况下,我列出了可用的具体实现,但其中许多是为我工作过的企业私下编写的。您可能需要编写自己的特定工具,但我鼓励您在可用时使用开源工具,并在可能的情况下对其做出贡献 。
Unfortunately, there is a dearth of freely available open source tooling for managing event-driven microservices. Where applicable, I have listed specific implementations that are available, but many of them have been privately written for the businesses I have worked for. You will likely need to write your own specific tools, but I encourage you to use open source tooling when available and to contribute back to it when possible.
当公司拥有少量系统时,很容易使用部落知识或非正式方法来跟踪谁拥有哪些系统。然而,在微服务世界中,显式跟踪微服务实现和事件流的所有权非常重要。通过遵循单写入器原则(请参阅“微服务单写入器原则”),您可以将事件流所有权归回拥有写入权限的微服务。
When a company has a small number of systems, it’s easy to use tribal knowledge or informal methods to keep track of who owns which systems. In the microservice world, however, it is important to explicitly track ownership of microservice implementations and event streams. By following the single writer principle (see “Microservice Single Writer Principle”), you can attribute event stream ownership back to the microservice that owns the write permissions.
您可以使用内部开发的简单微服务来跟踪和管理人员、团队和微服务之间的所有依赖关系。该系统是本章中许多其他工具的基础,因此我强烈建议您寻找或开发它。以这种方式分配微服务所有权有助于确保将细粒度的 DevOps 权限正确分配给需要它们的团队。
You can use a simple microservice developed in-house to track and manage all of the dependencies between people, teams, and microservices. This system is the foundation for many of the other tools within this chapter, so I strongly suggest that you look into finding or developing it. Assigning microservice ownership this way helps ensure that fine-grained DevOps permissions are correctly assigned to the teams that need them.
团队需要能够创建新的事件流并进行相应的修改。微服务应该有权自动创建自己的内部事件流,并完全控制重要属性,例如分区计数、保留策略和复制因子。
Teams will need the ability to create new event streams and modify them accordingly. Microservices should have the right to automatically create their own internal event streams and have full control over important properties such as the partition count, retention policy, and replication factor.
例如,包含在任何情况下都不会丢失的非常重要、极其敏感的数据的流可能具有无限保留策略和高复制因子。或者,包含大量单独不重要更新的流可能具有高分区计数、低复制因子和短保留策略。创建事件流时,通常会将其所有权分配给特定的微服务甚至外部系统。这将在下一节中介绍。
For instance, a stream that contains highly important, extremely sensitive data that cannot be lost under any circumstances may have an infinite retention policy and a high replication factor. Alternately, a stream containing a high volume of individually unimportant updates may have a high partition count with a low replication factor and a short retention policy. Upon creating an event stream, it is customary to assign ownership of it to a particular microservice or even an external system. This is covered in the next section.
分配所有权的一种有用技术是使用元数据标记流。然后,只有拥有流制作权限的团队才能添加、修改或删除元数据标签。有用元数据的一些示例包括但不限于以下内容:
One useful technique for assigning ownership is to tag streams with metadata. Then, only teams that own the production rights to a stream can add, modify, or remove metadata tags. Some examples of useful metadata include, but are not limited to, the following:
拥有流的服务。在传达更改请求或审核哪些流属于哪些服务时,经常使用此元数据。它使组织中任何微服务或事件流的所有权和业务通信结构更加清晰。
The service that owns a stream. This metadata is regularly used when communicating change requests or auditing which streams belong to which services. It adds clarity to ownership and the business communications structure of any microservice or event stream in your organization.
需要更严格安全处理的信息,因为它可以直接或间接识别用户。此元数据的基本用例之一是限制对标记为 PII 的任何事件流的访问,除非拥有数据的团队明确批准。
Information that requires stricter security handling because it can identify users either directly or indirectly. One of the basic use cases of this metadata is to restrict access to any event stream marked as PII unless the team owning the data explicitly gives approval.
任何与金钱、账单或其他重要创收事件有关的事情。与 PII 类似但不相同。
Anything pertaining to money, billing, or other important revenue-generating events. Similar but not identical to PII.
与业务的嵌套有界上下文结构对齐的描述符。分配有命名空间的流可以对命名空间外部的服务隐藏,但可用于命名空间内的服务。这有助于通过向浏览可用事件流的用户隐藏不可访问的事件流来减少数据发现过载。
A descriptor aligned with the nested bounded context structures of the business. A stream with a namespace assigned could be hidden from services outside of the namespace, but available for services within the namespace. This helps reduce data discovery overload by concealing inaccessible event streams to a user browsing through available event streams.
一种指示流已过时或因某种原因已被取代的方式。将事件流标记为已弃用允许旧系统继续使用它,同时阻止新的微服务请求订阅。当必须对现有事件流的数据格式进行重大更改时,通常使用此标记。新事件可以放入新流中,同时维护旧流,直到依赖的微服务可以迁移为止。最后,当不再有已弃用流的注册使用者时,可以通知已弃用事件流所有者,此时可以安全地删除它。
A way of indicating that a stream is outdated or has been superseded for some reason. Tagging an event stream as deprecated allows for grandfathered systems to continue using it while new microservices are blocked from requesting a subscription. This tag is generally used when breaking changes must be made to the data format of an existing event stream. The new events can be put into the new stream, while the old stream is maintained until dependent microservices can be migrated over. Finally, the deprecated event stream owner can be notified when there are no more registered consumers of the deprecated stream, at which point it may be safely deleted.
任何其他可能适合您业务的元数据都可以而且应该使用此工具进行跟踪。考虑哪些标签对您的组织可能很重要并确保它们可用。
Any other metadata that may be suitable to your business can and should be tracked with this tool. Consider which tags may be important to your organization and ensure they are available.
配额通常由事件代理在通用级别建立。例如,事件代理可能被设置为只允许 20% 的 CPU 处理时间用于服务单个生产者或消费者组。此配额可防止由于意外的闲聊生产者或从非常大的事件流开始时高度并行的消费者组而导致的意外拒绝服务。一般来说,您至少要确保整个集群不会因一项服务的 I/O 请求而饱和。您可以简单地限制消费者或生产者可以使用的资源数量,从而导致其受到限制。
Quotas are generally established by the event broker at a universal level. For instance, an event broker may be set to allow only 20% of its CPU processing time to go toward serving a single producer or consumer group. This quota prevents accidental denial of service due to an unexpectedly chatty producer or a highly parallelized consumer group beginning from the start of a very large event stream. In general, you want to ensure at the very least that your entire cluster won’t be saturated by one service’s I/O requests. You can simply limit how many resources a consumer or producer can use, resulting in it being throttled.
您可能需要更细粒度地设置配额,以防止容易出现激增的系统受到限制,同时仍确保稳定状态消费者的最小处理能力和网络 I/O。您可能需要为从事件代理集群外部的源生成数据的生产者设置不同的配额或完全删除它们。例如,如果生产者的生产速率低于传入消息速率,则基于第三方输入流或外部同步请求发布事件的生产者可能最终会丢弃数据或崩溃。
You may need to set up quotas at a more granular level, preventing surge-prone systems from being throttled while still ensuring a minimum amount of processing power and network I/O for steady-state consumers. You may want to set up different quotas or remove them entirely for producers producing data from sources outside the event broker cluster. For instance, a producer publishing events based on third-party input streams or external synchronous requests may simply end up dropping data or crashing if its production rate is throttled below the incoming message rate.
显式模式为事件建模提供了强大的框架。数据的精确定义(包括名称、类型、默认值和文档)为事件的生产者和消费者提供了清晰的信息。模式注册表是一项服务,允许您的生产者注册他们用于编写事件的模式。这提供了几个明显的好处:
Explicit schemas provide a strong framework for modeling events. Precise definitions of data, including names, types, defaults, and documentation, provide clarity to both producers and consumers of the event. The schema registry is a service that allows your producers to register the schemas they have used to write the event. This provides several distinct benefits:
事件模式不需要与事件一起传输。可以使用简单的占位符 ID,从而显着减少带宽使用。
The event schema does not need to be transported with the event. A simple placeholder ID can be used, significantly reducing bandwidth usage.
模式注册表提供了用于获取事件模式的单点参考。
The schema registry provides the single point of reference for obtaining the schemas for an event.
模式支持数据发现,尤其是自由文本搜索。
Schemas enable data discovery, particularly with free-text search.
The workflow for a schema registry is shown in Figure 14-1.
生产者在生产之前序列化事件时,向模式注册表注册模式以获得模式的 ID(步骤 1)。然后,它将 ID 附加到序列化事件(步骤 2),并将信息缓存在生产者缓存中(步骤 3),以避免再次查询注册表以获取该架构。请记住,生产者必须为每个事件完成此过程,因此消除对已知事件格式的外部查询至关重要。
The producer, upon serializing the event prior to production, registers the schema with the schema registry to obtain the schema’s ID (step 1). It then appends the ID to the serialized event (step 2) and caches the information in the producer cache (step 3) to avoid querying the registry again for that schema. Remember, the producer must complete this process for each event, so eliminating the external query for known event formats is essential.
使用者接收事件并从其缓存或架构注册表获取该特定 ID 的架构(步骤 4)。然后,它将 ID 换成架构(步骤 5),并将事件反序列化为已知格式。如果架构是新的,则会缓存该架构(步骤 6),并且消费者的业务逻辑现在可以使用反序列化的事件。在此阶段,事件还可以应用模式演化。
The consumer receives the event and gets the schema (step 4) for that specific ID from either its cache or the schema registry. It then swaps the ID out for the schema (step 5) and deserializes the event into the known format. The schema is cached if new (step 6), and the deserialized event can now be used by the consumer’s business logic. At this stage the event could also have schema evolution applied.
Confluence 为 Apache Kafka 提供了架构注册表的出色实现。它支持 Apache Avro、Protobuf 和 JSON 格式,并且可免费供生产使用。
Confluent has provided an excellent implementation of a schema registry for Apache Kafka. It supports Apache Avro, Protobuf, and JSON formats and is freely available for production use.
事件流模式对于标准化通信非常重要。可能出现的一个问题是,通知其他团队他们所依赖的模式已经演变(或将要演变)可能会出现问题,尤其是在存在大量事件流的情况下。这就是模式创建和修改通知发挥作用的地方。
Event stream schemas are important in terms of standardizing communication. One issue that can arise, particularly with large numbers of event streams, is that it can be problematic to notify other teams that a schema they depend on has evolved (or will be evolving). This is where schema creation and modification notifications come into play.
通知系统的目标只是在消费者的输入模式发生变化时向他们发出警报。访问控制列表(ACL,本章稍后讨论)是确定哪个微服务消耗哪个事件流以及通过关联确定它所依赖的模式的好方法。
The goal of a notification system is simply to alert consumers when their input schemas have evolved. Access control lists (ACLs, discussed later in this chapter) are a great way to determine which microservice consumes from which event stream and, by association, which schemas it depends on.
架构更新可以从架构流中使用(如果您使用 Confluence 架构注册表)并交叉引用其关联的事件流。从这里,ACL 提供有关哪些服务正在消耗哪些事件流的信息,然后通过微服务到团队分配系统通知拥有这些服务的相应团队。
Schema updates can be consumed from the schema stream (if you’re using the Confluent schema registry) and cross-referenced to their associated event streams. From here, the ACLs provide information about which services are consuming which event streams and then notify the corresponding teams that own the services via the microservice-to-team assignment system.
通知系统有很多好处。虽然在完美的世界中,每个消费者都能够全面审查模式的每个上游更改,但通知系统提供了一个安全网,可以在有害或破坏性更改变成危机之前识别它们。最后,消费者可能只想关注公司内所有公开可用的架构更改,以便他们在新事件流上线时能够更深入地了解数据。
There are a number of benefits to a notification system. While in a perfect world, every consumer would be able to fully review every upstream change to the schema, a notification system provides a safety net for identifying detrimental or breaking changes before they become a crisis. Lastly, a consumer may want to just follow all publicly available schema changes across a company, allowing them greater insight into the data as new event streams come online.
事件驱动的微服务要求您在继续数据处理之前管理偏移量。在正常操作中,微服务将在处理消息时提前其消费者偏移量。但是,在某些情况下您必须手动调整偏移量。
Event-driven microservices require that you manage offsets before they proceed with data processing. In normal operation, the microservice will advance its consumer offset as it processes messages. There are, however, cases where you’ll have to manually adjust the offset.
更改微服务的逻辑可能需要您重新处理之前时间点的事件。通常重新处理需要从流的开头开始,但您的选择点可能会根据您的服务需求而有所不同。
Changing the logic of the microservice may require that you reprocess events from a previous point in time. Usually reprocessing requires starting at the beginning of the stream, but your selection point may vary depending on your service’s needs.
或者,也许您的微服务不需要旧数据,而应该仅使用最新数据。您可以将应用程序偏移量重置为最新偏移量,而不是最早的偏移量。
Alternately, perhaps your microservice doesn’t need old data and should consume only the newest data. You can reset the application offset to be the latest offset, instead of the earliest.
您可能希望将偏移重置为特定时间点。这通常会在多集群故障转移中发挥作用,您希望确保没有错过任何消息,但又不想从头开始。一种策略包括将偏移量重置为崩溃前N分钟的时间,以确保不会丢失任何复制的消息。
You may want to reset the offset to a specific point in time. This often comes into play with multicluster failover, where you want to ensure you haven’t missed any messages but don’t want to start at the beginning. One strategy includes resetting the offset to a time N minutes prior to the crash, ensuring that no replicated messages are missed.
对于生产级 DevOps,团队必须拥有微服务才能修改其偏移量,这是微服务到团队分配系统提供的功能。
For production-grade DevOps, a team must own the microservice in order to modify its offsets, a feature provided by the microservice-to-team assignment system.
数据访问控制不仅从业务安全的角度来看很重要,而且作为执行单写入者原则的一种手段。权限和访问控制列表确保有界上下文可以强制执行其边界。对给定事件流的访问权限只能由拥有生产微服务的团队授予,您可以使用微服务到团队分配系统来强制执行这一限制。权限通常分为以下常见类别(当然,取决于事件代理实现):READ、WRITE、 CREATE、DELETE、MODIFY 和 DESCRIBE。
Access control to data is important not only from a business security standpoint, but also as a means of enforcing the single writer principle. Permissions and access control lists ensure that bounded contexts can enforce their boundaries. Access permissions to a given event stream should be granted only by the team that owns the producing microservice, a restriction you can enforce by using the microservice-to-team assignment system. Permissions usually fall into these common categories (depending, of course, on the event broker implementation): READ, WRITE, CREATE, DELETE, MODIFY, and DESCRIBE.
ACL 依赖于每个消费者和生产者的个人身份识别。确保尽快(最好从第一天开始)为事件代理和服务启用并强制实施身份识别。事后添加标识是非常痛苦的,因为它需要更新和检查连接到事件代理的每一个服务。
ACLs rely on individual identification for each consumer and producer. Ensure that you enable and enforce identification for your event broker and services as soon as possible, preferably from day one. Adding identification after the fact is extremely painful, as it requires updating and reviewing every single service that connects to the event broker.
ACL 强制执行有界上下文。例如,微服务应该是其内部和变更日志事件流的创建、写入和读取权限的唯一所有者。微服务在任何时候都不应耦合到另一个微服务的内部事件流。此外,根据单写入器原则,该微服务应该是唯一为其输出流分配写入权限的服务。输出流可以公开,以便任何其他系统都可以使用该数据,或者它可能具有受限的访问权限,因为它包含敏感的财务或 PII 数据或者是嵌套有界上下文的一部分。
ACLs enforce bounded contexts. For instance, a microservice should be the only owner of CREATE, WRITE, and READ permissions for its internal and changelog event streams. At no point should a microservice couple on the internal event streams of another microservice. Additionally, this microservice should be the only service assigned WRITE permissions to its output stream, according to the single writer principle. The output streams may be made publicly available such that any other system can consume the data, or it may have restricted access because it contains sensitive financial or PII data or is part of a nested bounded context.
典型的微服务将被单独分配一组遵循表 14-1所示格式的权限。
A typical microservice will be individually assigned a set of permissions following the format shown in Table 14-1.
| 成分 | 微服务权限 |
|---|---|
输入事件流 Input event streams |
读 READ |
输出事件流 Output event streams |
创建、写入(如果内部使用,可能还可以读取) CREATE, WRITE (and maybe READ, if used internally) |
内部和变更日志事件流 Internal and changelog event streams |
创建、写入、读取 CREATE, WRITE, READ |
一个特别有用的功能是为各个团队提供请求消费者访问特定微服务的方法,从而将执行访问控制的责任转移给他们。或者,根据业务需求和元数据标签,您可以集中此流程,以便团队在请求访问敏感信息时进行安全审查。权限的授予和撤销可以保留为自己的事件流,从而为审计目的提供持久且不可变的数据访问记录。
One particularly helpful feature is to provide individual teams with the means of requesting consumer access for a specific microservice, offloading the responsibilities of access control enforcement to them. Alternately, depending on business requirements and metadata tags, you could centralize this process so that teams go through a security review whenever requesting access to sensitive information. The granting and revoking of permissions can be kept as its own stream of events, providing a durable and immutable record of data access for auditing purposes.
当更改有状态应用程序的实现时,重置应用程序的内部状态是很常见的。对存储在内部和变更日志事件流中的数据结构的任何更改,以及对拓扑工作流的任何更改,都需要根据新应用程序删除并重新创建 流。
It is common to reset the internal state of the application when changing a stateful application’s implementation. Any changes to the data structures stored in the internal and changelog event streams, as well as any changes to the topology workflow, will require that the streams be deleted and re-created according to the new application.
第 7 章中讨论的一些有状态微服务模式使用处理节点外部的状态存储。根据您公司的微服务平台组织支持的功能,当微服务所有者请求时,可能(并且当然建议)重置这些外部状态存储。例如,如果微服务正在使用外部状态存储,例如 Amazon 的 DynamoDB 或 Google 的 Bigtable,则最好在重置应用程序时清除关联的状态。这减少了运营开销并确保自动删除任何陈旧或错误的数据。“官方支持的功能”范围之外的任何外部有状态服务可能都需要手动重置。
Some of the stateful microservice patterns discussed in Chapter 7 use state stores external to the processing node. Depending on the capabilities supported by your company’s microservice platform organization, it may be possible (and is certainly advisable) to reset these external state stores when requested by the microservice owner. For example, if a microservice is using an external state store, such as Amazon’s DynamoDB or Google’s Bigtable, it would be best to purge the associated state when resetting the application. This reduces operational overhead and ensures that any stale or erroneous data is automatically removed. Any external stateful services outside the domain of “officially supported capabilities” will likely need to be manually reset.
需要注意的是,虽然此工具应该是自助式的,但另一个团队决不能删除另一个团队拥有的事件流和状态。再次,我建议使用本章中讨论的微服务到团队分配系统,以确保应用程序只能由其所有者或管理员重置。
It’s important to note that while this tool should be self-serve, in no way should another team be able to delete the event streams and state owned by another team. Again, I recommend using the microservice-to-team assignment system discussed in this chapter to ensure that an application can be reset only by its owner or an admin.
总而言之,这个工具需要:
In summary, this tool needs to:
删除微服务的内部流和变更日志流
Delete a microservice’s internal streams and changelog streams
删除任何外部状态存储实现(如果适用)
Delete any external state store materializations (if applicable)
将消费者组偏移重置为每个输入流的开头
Reset the consumer group offsets to the beginning for each input stream
消费者滞后是事件驱动的微服务需要扩展的最佳指标之一。您可以使用定期计算相关消费者组滞后的工具来监控这一点。尽管代理实现之间的机制可能有所不同,但滞后的定义是相同的:给定微服务使用者组的最新事件和最后处理的事件之间的事件计数差异。滞后的基本测量(例如阈值测量)相当简单且易于实现。例如,如果消费者的偏移滞后大于M 个事件的N 个事件分钟,触发消费者处理器加倍并重新平衡工作负载。如果延迟得到解决并且当前运行的处理器数量高于所需的最低数量,请减少处理器计数。
Consumer lag is one of the best indicators that an event-driven microservice needs to scale up. You can monitor for this by using a tool that periodically computes the lag of consumer groups in question. Though the mechanism may vary between broker implementations, the definition of lag is the same: the difference in event count between the most recent event and the last processed event for a given microservice consumer group. Basic measurements of lag, such as a threshold measurement, are fairly straightforward and easy to implement. For instance, if a consumer’s offset lag is greater than N events for M minutes, trigger a doubling of consumer processors and rebalance the workload. If the lag is resolved and the number of processors currently running is higher than the minimum required, scale the processor count down.
一些监控系统(例如 Apache Kafka 的 Burrow)在计算滞后状态时会考虑偏移滞后的历史记录。如果有大量事件进入流,导致下一个事件到达之前的一瞬间延迟量仅为 0,则此方法非常有用。由于滞后测量本质上往往是周期性的,因此系统可能总是显得比传统测量滞后。因此,利用与历史规范的偏差可以成为确定系统是否落后或赶上的有用机制。
Some monitoring systems, such as Burrow for Apache Kafka, consider the history of offset lag when computing the lag state. This approach can be useful in cases where you have a large volume of events entering a stream, such that the amount of lag is only ever at 0 for a split second before the next event arrives. Since lag measurements tend to be periodic in nature, it is possible that the system will always appear to be lagging by a conventional measurement. Therefore, using deviation from historical norms can be a useful mechanism for determining if a system is falling behind or catching up.
请记住,虽然微服务应该可以根据需要自由扩展和缩小,但通常会使用某种形式的滞后(容忍阈值)来防止系统无休止地扩展和缩小。这种磁滞回线需要成为评估信号的逻辑的一部分,并且通常可以由 AWS CloudWatch 和 Google Cloud Operations(以前称为 Stackdriver)等现代云平台来容纳。
Remember that while microservices should be free to scale up and down as required, generally some form of hysteresis—a tolerance threshold—is used to prevent a system from scaling up and down endlessly. This hysteresis loop needs to be part of the logic that evaluates the signal and can often be accommodated by modern cloud platforms such as AWS CloudWatch and Google Cloud Operations (formerly Stackdriver).
为新的业务需求创建代码存储库是微服务环境中的典型任务。将此任务自动化为简化的流程将确保一切都可以组合在一起并集成到功能团队提供的通用工具中。
Creating a code repository for a new business requirement is a typical task in a microservice environment. Automating this task into a streamlined process will ensure that everything fits together and integrates into the common tooling provided by the capabilities teams.
以下是一个典型的微服务创建流程:
Here is a typical microservice creation process:
创建存储库。
Create a repository.
与持续集成管道创建任何必要的集成(在“持续集成、交付和部署系统”中讨论)。
Create any necessary integrations with the continuous integration pipeline (discussed in “Continuous Integration, Delivery, and Deployment Systems”).
配置任何 Webhook 或其他依赖项。
Configure any webhooks or other dependencies.
使用微服务到团队分配系统将所有权分配给团队。
Assign ownership to a team using the microservice-to-team assignment system.
注册输入流的访问权限。
Register for access permissions from input streams.
创建任何输出流并应用所有权权限。
Create any output streams and apply ownership permissions.
提供应用模板或代码生成器来创建微服务框架的选项。
Provide the option for applying a template or code generator to create the skeleton of the microservice.
团队将多次完成此流程,因此以这种方式简化流程将节省大量时间和精力。新的自动化工作流程包括最新模板和代码生成器的注入点,确保新项目包含最新支持的代码和工具,而不是简单地复制旧项目。
Teams will complete this process many times over, so streamlining it in this way will save significant time and effort. The newly automated workflow includes an injection point for up-to-date templates and code generators, ensuring that new projects include the latest supported code and tools instead of simply copying an older project.
容器管理由容器管理服务 (CMS) 处理,如第 2 章所述。我建议公开 CMS 的某些方面,以便团队可以提供自己的 DevOps 功能,例如:
Container management is handled by the container management service (CMS), as discussed in Chapter 2. I recommend exposing certain aspects of the CMS so teams can provide their own DevOps capabilities, such as:
为其微服务设置环境变量
Setting environment variables for their microservices
指示在哪个集群上运行微服务(例如,测试、集成、 生产)
Indicating which cluster to run a microservice on (e.g., testing, integration, production)
根据微服务的需求管理 CPU、内存和磁盘资源
Managing CPU, memory, and disk resources, depending on the needs of their microservices
手动增加和减少服务计数,或根据服务级别协议和处理延迟
Increasing and decreasing service count manually, or depending on service-level agreements and processing lag
自动缩放 CPU、内存、磁盘或滞后指标
Autoscaling on CPU, memory, disk or lag metrics
企业需要确定应向开发人员公开多少容器管理选项,以及应由专门的运营团队管理多少容器管理选项。这通常取决于组织内的 DevOps 文化 。
The business will need to determine how many container management options should be exposed to developers, versus how many should be managed by a dedicated operations team. This typically depends on the culture of DevOps within the organization.
随着公司围绕事件驱动的微服务进行扩展,集群的创建和管理往往会出现。一般来说,中小型公司通常可以使用单个事件代理集群来满足其所有服务需求。然而,由于各种技术和法律原因,较大的公司经常发现自己面临提供多个集群的压力。国际公司可能需要将某些数据保留在原籍国。尽管现代事件代理具有出色的水平扩展质量,但数据大小可能会增长得如此之大,以至于实际上无法将其全部保存在单个集群中。组织中的各个业务部门可能需要自己的集群来实现隔离目的。也许最常见的是,
Cluster creation and management tends to come up as a company scales around event-driven microservices. Generally speaking, a small to medium-sized company can often get away with using a single event broker cluster for all of its serving needs. However, larger companies often find themselves under pressure to provide multiple clusters for various technical and legal reasons. International companies may need to keep certain data within the country of origin. Data sizes may grow so large that it cannot all be practically kept within a single cluster, despite modern event brokers’ excellent horizontal scaling qualities. Various business units in an organization may require their own clusters for isolation purposes. Perhaps most commonly, data must be replicated across multiple clusters in multiple regions to provide redundancy in case of a total cluster outage.
多集群管理,包括动态跨区域通信和灾难恢复,是一个复杂的主题,完全可以写成一本书。它还高度依赖于相关服务以及所使用的预防和恢复策略。一些企业(例如 Capital One)拥有围绕 Apache Kafka 实现构建的重要自定义库和代码,以允许本机多集群复制。作为一家银行,公司不能承受任何金融交易事件的损失。您的需求可能会有所不同。由于这些原因,本书不涉及多集群服务和数据管理策略。
Multicluster management, including dynamic cross-region communication and disaster recovery, is a complex topic that could very well fill its own book. It is also highly dependent on the services in question and the prevention and recovery strategies being used. Some businesses, like Capital One, have significant custom libraries and code built around their Apache Kafka implementations to allow for native multicluster replication. As a bank, the company cannot afford to lose any financial transaction events whatsoever. Your needs may vary. For these reasons, this book doesn’t cover multicluster service and data management strategies.
负责管理事件代理集群的团队通常还会提供用于创建和管理新集群的工具。话虽如此,商业云提供商也正在进入这个领域。例如,Apache Kafka 集群现在可以在 AWS 中按需创建(截至 2018 年 11 月),加入许多其他云服务提供商的行列。不同的事件代理技术可能需要不同数量的工作来支持,并且应该由您的领域专家仔细检查。无论哪种情况,目标都是拥有一个事件代理集群管理工具,整个组织可以使用它来轻松创建和扩展事件代理。
The team responsible for managing the event broker clusters will often also provide tooling for creating and managing new clusters. That being said, commercial cloud providers are also moving into this domain; for instance, Apache Kafka clusters can now be created on-demand in AWS (as of November 2018), joining a number of other cloud service providers. Different event broker technologies may require varying amounts of work to support and should be examined closely by your domain experts. In either case, the goal is to have an event broker cluster management tool that the entire organization can use to easily create and scale event brokers.
您经常需要启动一组独立于所有其他资源的计算资源。并不总是需要创建全新的容器管理服务,因为现有的容器管理服务通常可以服务多个命名空间。与事件代理一样,云计算提供商通常提供托管服务,可以按需为您提供这些功能,例如 Google 和 Amazon 的托管 Kubernetes 解决方案。
You’ll often need to bring up a set of compute resources that are independent of all other resources. It is not always necessary to create an entirely new container management service, as usually the existing one can serve multiple namespaces. As with event brokers, cloud computing providers commonly provide hosted services that can give you these capabilities on-demand, such as Google’s and Amazon’s hosted Kubernetes solutions.
适用于事件代理的相同技术和法律要求也适用于计算资源。通过跨数据中心分配处理来避免区域故障,在无法离开该国家/地区的情况下在本地处理数据,并通过动态地将计算密集型工作负载转移到更便宜的服务提供商来节省资金。
The same technical and legal requirements that apply to event brokers extend to compute resources. Avoid regional failures by distributing processing across data centers, process data locally if it cannot leave the country, and save money by dynamically shifting compute-heavy workloads to cheaper service providers.
通常,您可以使用相同的持续集成和持续交付(CI 和 CD)工具来执行此任务,但您将需要一种选择机制来确定在何处部署微服务。此外,您需要确保所需的事件数据可供计算资源使用,通常是通过同一区域或可用区中的托管来实现。跨区域通信始终是可能的,但往往成本高昂且速度缓慢。
Generally you can use the same continuous integration and continuous delivery (CI and CD) tools to perform this task, but you will need a selection mechanism to determine where to deploy the microservices. Additionally, you will need to ensure that the required event data is available to the compute resources, generally through colocation within the same region or availability zone. Cross-region communication is always possible, but it tends to be expensive and slow.
在集群之间复制事件数据对于将事件驱动的微服务扩展到单个集群之外非常重要,例如用于灾难恢复、定期跨集群通信和以编程方式生成的测试环境。
Replicating event data between clusters is important for scaling up event-driven microservices beyond the confines of a single cluster—examples include for the purposes of disaster recovery, regular cross-cluster communication, and programmatically generated testing environments.
集群之间复制数据的具体方式因事件代理和复制工具实现而异。选择复制工具实现时,请考虑以下因素:
The specifics of how data is replicated between clusters vary with event broker and replication tool implementations. When selecting a replication tool implementation, consider the following:
它会自动复制新添加的事件流吗?
Does it replicate newly added event streams automatically?
它如何处理已删除或修改的事件流的复制?
How does it handle the replication of deleted or modified event streams?
数据是使用相同的偏移量、分区和时间戳精确复制还是近似复制?
Is the data replicated exactly, with the same offsets, partitions, and timestamps, or approximately?
复制的延迟是多少?是否能够满足业务需求?
What is the latency in replication? Is it acceptable to the business needs?
有哪些性能特点?能否根据业务需求进行扩展?
What are the performance characteristics? Can it scale according to business needs?
最后但同样重要的一点是,到目前为止讨论的相同工具集也应该以编程方式为新集群提供。这提供了一组通用工具,您可以将其部署到任何集群部署,而无需依赖除事件代理本身之外的任何数据存储。以这种方式使用工具有很多好处。首先,该工具的使用频率更高,有助于发现错误或需要添加的必要功能。其次,它降低了使用新集群的进入门槛,因为用户已经熟悉了工具界面。最后,当集群终止时,工具可以与其一起终止,无需额外的清理。
Last, but certainly not least, the same sets of tools discussed thus far should also be programmatically brought up for new clusters. This provides a common set of tools you can deploy to any cluster deployment without relying on any data stores besides the event broker itself. There are a number of benefits to using tooling in this manner. First, the tooling gets used far more often and helps reveal bugs or necessary features to be added. Second, it lowers the barrier to entry for using new clusters, as users will already be familiar with the tooling interfaces. Lastly, when the cluster is terminated, the tools can be terminated alongside it with no additional cleanup required.
跟踪微服务之间的数据依赖关系对于帮助运行事件驱动的微服务组织非常有用。唯一的要求是组织必须知道哪些微服务正在读取和写入哪些事件流。为了实现这一目标,它可以采用自我报告系统,让消费者和生产者报告自己的消费和生产模式。然而,任何形式的自我报告解决方案的问题在于,它实际上是自愿的,并且总会有许多团队忘记、选择退出或根本不愿意报告。在没有完全合规的情况下确定依赖关系并不是特别有用,因为通信结构中的差距和不完整的拓扑限制了洞察力。这就是本章前面讨论的权限结构和 ACL 发挥作用的地方。
Tracking the data dependencies between microservices can be extremely useful in helping run an event-driven microservice organization. The only requirement is that the organization must know which microservices are reading and writing to which event streams. To achieve this, it could employ a self-reporting system where consumers and producers report on their own consumption and production patterns. The problem with any sort of self-reporting solution, however, is that it is effectively voluntary, and there will always be a number of teams that forget, opt out, or simply are unwilling to report. Determining dependencies without full compliance is not particularly useful, as gaps in the communication structure and incomplete topologies limit insight. This is where the permissions structure and ACLs discussed earlier in this chapter come into play.
利用权限结构来确定依赖关系可以保证两件事。首先,如果不注册其权限要求,微服务就无法运行,因为它无法读取或写入任何事件流。其次,如果对权限结构进行任何更改,则用于确定依赖性和拓扑生成的关联权限也会更新。无需进行其他更改即可确保正确的依赖性跟踪。
Leveraging the permissions structure to determine dependencies guarantees two things. First, a microservice cannot operate without registering its permissions requirements, as it would not be able to read from or write to any event streams. Second, if any changes are made to the permissions structure, the associated permissions used to determine dependencies and topology generation are also updated. No other changes are required to ensure proper dependency tracking.
以下是此类工具的一些其他用途:
Here are some other uses of such a tool:
数据科学家和数据工程师经常遇到的一个问题是如何确定数据来自哪里以及它走了哪条路线。通过权限结构的完整图表,他们可以识别每个祖先服务和任何给定事件的流。这可以帮助他们将错误和缺陷追溯到源头,并确定给定数据转换中涉及的所有服务。请记住,可以在权限事件流和微服务到团队分配事件流中及时返回,以生成该时间点的拓扑视图。当您审核旧数据时,这通常非常有用。
One problem that data scientists and data engineers regularly encounter is how to determine where data came from and which route it took. With a full graph of the permissions structure, they can identify each ancestor service and stream of any given event. This can help them trace bugs and defects back to the source and determine all services involved in a given data transformation. Remember that it is possible to go back in time in the permissions event stream and the microservice-to-team assignment event streams to generate a view of the topology at that point in time. This is often quite useful when you are auditing old data.
拥有微服务和流的团队可以映射到拓扑上。当使用适当的可视化工具进行渲染时,拓扑将清楚地显示哪些团队直接负责哪些服务。
The teams owning the microservices and streams can be mapped onto the topology. When rendered with a proper visualization tool, the topology will clearly show which teams are directly responsible for which services.
可视化工具是数据发现的有用工具。潜在消费者可以看到哪些流可用以及它们的生产者和消费者是谁。如果需要有关流数据的更多信息,潜在消费者可以联系生产者。
Visualizers are a useful tool for data discovery. A prospective consumer can see which streams are available and who their producers and consumers are. If more information is needed about the stream data, the prospective consumer can contact the producers.
正如高度内聚和松散耦合的微服务是理想的一样,对于团队来说也是如此。有了这个工具,团队就可以测量微服务之间有多少内部连接以及它有多少跨界连接。一般的经验法则是外部连接越少越好;但简单的连接计数是一个非常基本的指标。然而,即使基本指标的一致应用也可以揭示团队之间的相对相互依赖性。
Just as it is ideal for microservices to be highly cohesive and loosely coupled, so too it is for teams. With this tooling in place, a team can measure how many internal connections between microservices and how many cross-boundary connections it has. A general rule of thumb is that the fewer external connections, the better; but a simple count of connections is a very basic metric. However, even a consistent application of a basic metric can reveal the relative interdependence between teams.
根据业务需求调整微服务可以将实施映射到业务需求。合理的做法是,在存储库README或微服务元数据存储中,与其代码一起明确说明每个微服务的业务需求。反过来,这可以映射到所属团队。
Aligning microservices along business requirements enables a mapping of implementation to business requirement. It is reasonable to explicitly state each microservice’s business requirements alongside its code, perhaps in the repository README or in the microservice metadata store. In turn, this can be mapped to the owning teams.
企业主可以查看此叠加层并问自己:“此实施结构是否符合该团队的目标和优先事项?” 这是企业可以使用的最重要的工具之一,可确保其技术团队与业务沟通结构保持一致。
A business owner could look at this overlay and ask themselves, “Does this implementation structure align with the goals and priorities of this team?” This is one of the most important tools a business can have at its disposal to ensure that its technical teams are aligned with the business communication structure.
图 14-2显示了包含 25 个微服务的拓扑,覆盖了四个团队的所有权。为了清楚起见,每个箭头代表事件流的数据生成以及消费进程的消费。因此,微服务 3 正在使用来自微服务 4 的数据流。
Figure 14-2 shows a topology with 25 microservices, overlaid with the ownership of four teams. For purposes of clarity, each arrow represents the production of data to an event stream as well as consumption by the consuming process. Thus, microservice 3 is consuming a stream of data from microservice 4.
该映射显示团队 2 负责两个微服务,这两个微服务不属于其主要有界上下文(右下)。如果团队 2 的业务目标与微服务 2 和 7 所提供的功能不一致,这可能会引起关注。此外,微服务 2 和 7 对团队 1、3 和 4 都有一定的依赖性,这会增加团队 2 向外界暴露的“表面积”。互连性的衡量标准如表 14-2所示。
The mapping shows that team 2 is responsible for two microservices that are not part of its main bounded context (bottom right). This may be of concern if the business goals of team 2 do not align with the functions being served by microservices 2 and 7. Additionally, both microservices 2 and 7 have a number of dependencies on teams 1, 3, and 4, which increases the “surface area” that team 2 exposes to the outside world. A measure of interconnectedness is shown in Table 14-2.
| 传入流 | 传出流 | 传入的团队连接 | 传出的团队联系 | 拥有的服务 | |
|---|---|---|---|---|---|
1队 Team 1 |
1 1 |
3 3 |
1(第二队) 1 (Team 2) |
2(2,3队) 2 (Teams 2,3) |
5 5 |
2队 Team 2 |
8 8 |
2 2 |
3(1、3、4 队) 3 (Teams 1,3,4) |
2(1,4队) 2 (Teams 1,4) |
8 8 |
第三队 Team 3 |
3 3 |
3 3 |
2(1,4队) 2 (Teams 1,4) |
1(第二队) 1 (Team 2) |
6 6 |
第4队 Team 4 |
1 1 |
5 5 |
1(第 1 队) 1 (Team 1) |
2(2,3队) 2 (Teams 2,3) |
8 8 |
让我们看看如果减少团队间连接的数量以及团队边界传入和传出流的数量会发生什么。微服务 2 和 7 是重新分配的良好候选者,仅仅是因为它们在拓扑中的所有权孤岛,并且可以重新分配以减少跨团队依赖的数量。微服务7可以分配给团队1(或团队3),微服务2可以分配给团队4。现在更明显的是,微服务1也可以分配给团队4,以进一步减少跨界通信。结果如图14-3和表14-3所示。
Let’s see what happens if we reduce the number of interteam connections and the number of incoming and outgoing streams at the team boundary. Microservices 2 and 7 are good candidates for being reassigned simply due to their ownership island in the topology, and can be reassigned to reduce the number of cross-team dependencies. Microservice 7 can be assigned to team 1 (or to team 3), and microservice 2 can be assigned to team 4. It is also more apparent now that microservice 1 may also be assigned to team 4 to further reduce the cross-boundary communication. This result is shown in Figure 14-3 and Table 14-3.
| 传入流 | 传出流 | 传入的团队连接 | 传出的团队联系 | 拥有的服务 | |
|---|---|---|---|---|---|
1队 Team 1 |
1 1 |
3 3 |
1(第 3 队) 1 (Team 3) |
1(第 3 队)(–1) 1 (Team 3) (–1) |
5 5 |
2队 Team 2 |
4 (–4) 4 (–4) |
0 (–2) 0 (–2) |
2(3,4 队)(–1) 2 (Teams 3,4) (–1) |
0 (–2) 0 (–2) |
6 (–2) 6 (–2) |
第三队 Team 3 |
3 3 |
3 3 |
2(1,4队) 2 (Teams 1,4) |
2(球队 1,2)(+1) 2 (Teams 1,2) (+1) |
6 6 |
第4队 Team 4 |
1 1 |
3 (–2) 3 (–2) |
1(第 1 队) 1 (Team 1) |
2(3,4队) 2 (Teams 3,4) |
8(+2) 8 (+2) |
计算跨边界依赖性有一个积极的结果:跨团队传入和传出流计数减少,团队之间净减少三个连接。最大限度地减少跨界连接的数量有助于优化微服务对团队的分配。然而,这不足以确定微服务向团队的分配。您还必须考虑各种因素,例如团队人数、专业领域和实施复杂性。
Computing the cross-boundary dependencies has a positive result: cross-team incoming and outgoing stream counts are reduced, with a net decrease of three connections between teams. Minimizing the number of cross-boundary connections helps optimize the assignment of microservices to teams. This is not sufficient, however, for determining the assignment of microservices to a team. You must also take into account a variety of factors, such as the team’s head count, areas of expertise, and implementation complexities.
但最重要的是,您必须考虑微服务正在执行的业务功能。考虑这样一个场景:一个团队生成大量事件数据,这些数据可能是从多个外部源消耗的。该团队的业务职责可能仅限于获取数据并将其组织到事件中,业务逻辑由其他消费者在下游执行。在这种情况下,团队将拥有许多流连接和团队连接。这是能够查看与团队拥有的微服务关联的业务功能的有用之处。
Most importantly of all, though, you must account for the business functions that a microservice is performing. Consider a scenario where one team produces a whole host of event data, perhaps consumed from a number of external sources. It could be that the business responsibility of that team is limited simply to sourcing and organizing the data into events, with business logic being performed downstream by other consumers. In this case, the team would have many stream connections and team connections. This is where it is useful to be able to view the business functions associated with the microservices owned by the team.
在前面的示例中,有许多问题值得提出。其一,微服务 2 的业务功能实现是否更接近团队 2 或团队 4 的目标?微服务 7 怎么样:它是否比团队 2 更符合团队 1 的目标?一般来说,哪些服务最适合哪个团队?这些答案往往是定性的,必须根据团队的目标仔细评估。团队目标随着业务需求的发展而变化是很自然的,确保分配给这些团队的微服务与总体业务目标保持一致非常重要。该工具可以深入了解这些分配,并有助于为企业主提供清晰的信息。
In the preceding example, there are a number of questions worth asking. For one, do the business function implementations of microservice 2 align closer to the goals of team 2 or team 4? What about microservice 7: is it aligned closer with team 1’s goals than team 2’s? And in general, which services align best with which team? These answers tend to be qualitative and must be carefully evaluated against the goals of the team. It is natural that team goals change as business needs evolve, and it is important to ensure that the microservices assigned to those teams align with the overarching business goals. This tooling provides insight into these allocations and helps provide clarity to business owners.
拥有多个自治服务需要您仔细考虑如何管理这些系统。本章描述的工具旨在帮助您的组织管理其服务。
Having multiple autonomous services requires that you carefully consider how you’re going to manage these systems. The tools described in this chapter are designed to help your organization manage its services.
随着服务数量的增加,任何人了解一切如何运作以及每项服务和流如何适应整体情况的能力都会减弱。明确跟踪服务和流所有权、模式和元数据等属性使组织能够组织和跟踪随时间的变化。减少部落知识并正式编码原本简单的流和服务属性应该是您的组织的优先事项。毕竟,对于创建的每个附加实例,服务或流属性的任何模糊性都会被放大。
As the number of services increases, the ability of any one person to know how everything works and how each service and stream fit into the big picture diminishes. Keeping explicit track of properties such as service and stream ownership, schemas, and metadata allows organizations to organize and track change across time. Reducing tribal knowledge and formally codifying otherwise simple stream and service properties should be priorities for your organization. After all, any ambiguity around the properties of a service or stream will be amplified for each additional instance created.
对团队服务的自治和控制是大规模管理微服务的重要方面。根据 DevOps 原则,您应该能够重置消费者组偏移量和微服务状态存储。
Autonomy and control over your team’s services are important aspects of managing microservices at scale. In accordance with DevOps principles, you should be able to reset consumer group offsets and microservice state stores.
图式促进对事件含义的共同理解。模式注册表提供了一种简化模式管理的机制,并且可用于通知感兴趣的各方有关特定模式的任何更改。
Schemas foster a common understanding of the meaning of events. The schema registry provides a mechanism for streamlining schema management and can be used to notify interested parties about any changes to a particular schema.
测试事件驱动的微服务的好处之一是它们非常模块化。服务的输入由事件流或来自请求-响应 API 的请求提供。状态被具体化到其自己的独立状态存储中,并且输出事件被写入服务的输出流。微服务的小型和专用特性使其比更大、更复杂的服务更容易测试。移动部件较少,处理 I/O 和状态的方法相对标准,并且有大量机会与其他微服务重用测试工具。本章介绍测试原理和策略,包括单元测试、集成测试和性能测试。
One of the great things about testing event-driven microservices is that they’re very modular. Input to the service is provided by event streams or by requests from a request-response API. State is materialized to its own independent state store, and output events are written to a service’s output streams. Microservices’ small and purpose-built nature make them far easier to test than larger and more complex services. There are fewer moving parts, a relatively standard methodology of handling I/O and state, and plenty of opportunity to reuse testing tooling with other microservices. This chapter covers testing principles and strategies, including unit testing, integration testing, and performance testing.
事件驱动的微服务共享所有应用程序通用的测试最佳实践。功能测试,例如单元测试、集成测试、系统测试和回归测试,确保微服务做它应该做的事情,而不做它不应该做的事情。非功能测试,例如性能、负载、压力和恢复测试,可确保其在各种环境场景下表现符合预期。
Event-driven microservices share the testing best practices that are common to all applications. Functional testing, such as unit, integration, system, and regression testing, ensures that the microservice does what it is supposed to and that it doesn’t do what it should not do. Nonfunctional testing, such as performance, load, stress, and recovery testing, ensures that it behaves as expected under various environmental scenarios.
现在,在进一步讨论之前,需要注意的是,本章旨在成为有关测试原理和操作方法的更广泛著作的姊妹篇。毕竟,许多书籍、博客和文档都是关于测试的,而我当然无法像他们那样涵盖测试。本章主要介绍事件驱动的特定测试方法和原则,以及它们如何集成到整体测试图中。请查阅您自己的有关特定于语言的测试框架和测试最佳实践的资料来补充本章。
Now, before going much further, it’s important to note that this chapter is meant to be a companion to more extensive works on the principles and how-tos of testing. After all, many books, blogs, and documents have been written on testing, and I certainly can’t cover testing to the extent that they do. This chapter primarily looks at event-driven-specific testing methodologies and principles and how they integrate into the overall testing picture. Consult your own sources on language-specific testing frameworks and testing best practices to complement this chapter.
单元测试用于测试应用程序中最小的代码片段,以确保它们按预期工作。这些小型测试单元为编写更大、更全面的测试来测试应用程序的更高功能奠定了基础。事件驱动的拓扑通常将转换、聚合、映射和归约函数应用于事件,使这些函数成为单元测试的理想选择。
Unit tests are used to test the smallest pieces of code in an application to ensure that they work as expected. These small testing units provide a foundation for which larger, more comprehensive tests can be written to test higher functionality of the application. Event-driven topologies often apply transformation, aggregation, mapping, and reduction functions to events, making these functions ideal candidates for unit testing.
确保测试每个函数的边界条件,例如空值和最大值。
Make sure to test boundary conditions, such as null and maximum values, for each of your functions.
无状态函数不需要先前函数调用的任何持久状态,因此很容易独立测试。以下代码显示了一个 EDM 拓扑示例,类似于您在 map-reduce-style 框架中找到的拓扑:
Stateless functions do not require any persistent state from previous function calls, and so are quite easy to test independently. The following code shows an example of an EDM topology similar to one that you would find in a map-reduce-style framework:
myInputStream.filter(myFilterFunction).map(myMapFunction).to(outputStream)
myInputStream.filter(myFilterFunction).map(myMapFunction).to(outputStream)
myMapFunction和myFilterFunction是独立的函数,两者都不保存状态。每个函数都应该经过单元测试,以确保它正确处理预期输入数据的范围,特别是极端情况。
myMapFunction and myFilterFunction are independent functions, neither of which keeps state. Each function should be unit-tested to ensure that it correctly handles the range of expected input data, particularly corner cases.
有状态函数的测试通常比无状态函数更复杂。状态可能会随着时间和输入事件的变化而变化,因此您必须小心测试所有必要的有状态边缘情况。有状态单元测试还要求持久状态(无论是模拟外部数据存储还是临时内部数据存储的形式)在测试期间可用。
Stateful functions are generally more complicated to test than stateless ones. State can vary with both time and input events, so you must take care to test all of the necessary stateful edge cases. Stateful unit testing also requires that persistent state, whether in the form of a mocked external data store or a temporary internal data store, is available for the duration of the test.
下面是一个有状态聚合函数的示例,可以在基本的生产者-消费者实现中找到:
Here is an example of a stateful aggregation function that might be found in a basic producer-consumer implementation:
publicLongaddValueToAggregation(Stringkey,LongeventValue){//The data store needs to be made available to the unit-test environmentLongstoredValue=datastore.getOrElse(key,0L);//Sum the values and load them back into the state storeLongsum=storedValue+eventValue;datastore.upsert(key,sum);returnsum;}
publicLongaddValueToAggregation(Stringkey,LongeventValue){//The data store needs to be made available to the unit-test environmentLongstoredValue=datastore.getOrElse(key,0L);//Sum the values and load them back into the state storeLongsum=storedValue+eventValue;datastore.upsert(key,sum);returnsum;}
该函数用于eventValue对每个键的所有 s 求和。模拟端点是在测试期间提供可靠的数据存储实现的一种方法。另一种选择是创建数据存储的本地可用版本,尽管这更类似于集成测试,稍后将更详细地介绍。无论哪种情况,您都必须仔细考虑此数据存储需要做什么以及它与运行时使用的实际实现有何关系。模拟往往效果很好,因为它允许非常高性能的单元测试,而不会受到旋转数据存储的完整实现的开销的负担。
This function is used to sum all eventValues for each key. Mocking the endpoint is one way of providing a reliable implementation of the data store for the duration of the test. Another option is creating a locally available version of the data store, though this is more akin to integration testing, which will be covered in more detail shortly. In either case, you must carefully consider what this data store needs to do and how it relates to the actual implementation used at runtime. Mocking tends to work well, because it allows for very high-performance unit testing that isn’t burdened by the overhead of spinning up a full implementation of the data store.
全功能的轻量级和重量级框架通常提供本地测试整个拓扑的方法。如果您的框架没有,用户和贡献者社区可能已经创建了提供此功能的第三方选项(这是选择具有强大社区的框架的另一个原因)。例如,Apache Spark除了提供用于对流输入和输出进行细粒度控制的内置类之外,还有两个独立的第三方单元测试选项StreamingSuiteBase和Spark-fast-tests 。Apache Flink 提供了自己的拓扑测试选项, Apache Beam也是如此。对于轻量级流框架,Kafka Streams 提供了使用 测试拓扑的方法,它可以模拟框架的功能,而无需您设置整个事件代理。MemoryStreamTopologyTestDriver
The full-featured lightweight and heavyweight frameworks typically provide the means to locally test your entire topology. If your framework does not, the community of users and contributors may have created a third-party option that provides this functionality (this is another reason to choose a framework with a strong community). For example, Apache Spark has two separate third-party options for unit tests, StreamingSuiteBase and spark-fast-tests, in addition to providing a built-in MemoryStream class for fine-grained control over stream input and output. Apache Flink provides its own topology testing options, as does Apache Beam. As for lightweight stream frameworks, Kafka Streams provides the means to test topologies using the TopologyTestDriver, which mocks out the functionality of the framework without requiring you to set up an entire event broker.
拓扑测试比单个单元测试更复杂,并根据业务逻辑指定测试整个拓扑。您可以将拓扑视为具有许多移动部件的单个、大型、复杂的函数。拓扑测试框架允许您完全控制向输入流生成哪些事件以及创建这些事件的时间。您可以生成具有特定值的事件;事件无序、包含无效时间戳或包含无效数据;以及用于练习极端情况逻辑的事件。通过这样做,您可以确保基于时间的聚合、事件调度和有状态函数等操作按预期执行。
Topology testing is more complex than a single unit test and exercises the entire topology as specified by your business logic. You can think of your topology as a single, large, complex function with many moving parts. The topology testing frameworks allow you to fully control which events are produced to the input streams as well as when they are created. You can generate events with specific values; events that are out of order, contain invalid timestamps, or include invalid data; and events that are used to exercise corner-case logic. By doing so, you can ensure that operations such as time-based aggregations, event scheduling, and stateful functions perform as expected.
例如,考虑以下映射缩减样式拓扑定义:
For example, consider the following map-reduce-style topology definition:
myInputStream.map(myMapFunction).groupByKey().reduce(myReduceFunction)
myInputStream.map(myMapFunction).groupByKey().reduce(myReduceFunction)
在此拓扑中,消耗的事件由变量 表示myInputStream。应用映射函数,将结果按键分组在一起,最后将它们简化为每个键的单个事件。myMapFunction虽然可以为和实现单元测试,但它们不能轻松地重现、和 的myReduceFunction框架操作,因为这些操作(以及其他操作)本质上是框架的一部分。mapgroupByKeyreduce
In this topology, consumed events are represented by the variable myInputStream. A mapping function is applied, the results are grouped together by key, and then finally they are reduced into a single event per key. While unit tests can be implemented for myMapFunction and myReduceFunction, they can’t easily reproduce the framework operations of map, groupByKey, and reduce, as these operations (among others) are inherently part of the framework.
这就是拓扑测试发挥作用的地方。每个流框架对测试拓扑都有不同级别的支持,您必须探索可用的选项。这些测试框架不需要您创建事件代理来保存输入事件或设置重量级框架集群进行处理。
This is where topology testing comes into play. Each stream framework has varying levels of support for testing the topology, and you must explore the options available to you. These testing frameworks do not require you to create an event broker to hold input events or to set up a heavyweight framework cluster for processing.
为了确保任何输出模式根据任何事件流模式演化规则与以前的模式兼容(请参阅“全功能模式演化”),您可以从模式注册表中提取模式并作为代码的一部分执行演化规则检查提交过程。某些应用程序可能会使用模式生成工具在编译时从代码中定义的类或结构自动生成模式,从而生成可与以前版本进行比较的以编程方式生成的模式。
To ensure that any output schemas are compatible with previous schemas according to any event stream schema evolutionary rules (see “Full-Featured Schema Evolution”), you can pull in the schemas from the schema registry and perform evolutionary rule checking as part of the code submission process. Some applications may use schema-generating tools to automatically generate the schemas from the classes or structs defined in the code at compile time, resulting in a programmatically generated schema that can be compared to previous versions.
微服务集成测试主要有两种类型:本地集成测试(测试在生产环境的本地副本上执行)和远程集成测试(微服务在本地系统外部的环境中执行)。这些解决方案都有许多优点和缺点,我们将很快探讨。
Microservice integration testing comes in two main flavors: local integration testing, where testing is performed on a localized replica of the production environment, and remote integration testing, where the microservice is executed on an environment external to the local system. Each of these solutions has a number of advantages and disadvantages, which we will explore shortly.
第三种风格是混合选项,其中微服务及其测试环境的某些部分在本地托管或执行,而其他部分则远程完成。由于从技术上讲不可能评估这种混合模式的所有组合和排列,因此我将只关注两种主要情况,并让您自行确定您自己的要求(如果它们不同)。
A third flavor is a hybrid option, where certain parts of your microservice and its test environment are hosted or executed locally and others done remotely. Since it’s technically impossible to evaluate all combinations and permutations of this hybrid pattern, I’ll just focus on the two main cases and leave it up to you to determine your own requirements should they differ.
在本章的其余部分中,您应该牢记几个首要问题:
There are several overarching questions that you should keep in mind for the remainder of this chapter:
您希望从集成测试中得到什么?是否像“这能运行吗?”那么简单。是用生产数据进行冒烟测试吗?或者是否有更复杂的工作流程需要测试和验证?
What are you hoping to get out of your integration testing? Is it as simple as “does this run?” Is it a smoke test with production data? Or are there more complex workflows that need to be tested and validated?
您的微服务是否需要支持从输入流时间开始重新启动,例如由于错误而导致数据全部丢失或重新处理的情况?如果是这样,您需要了解什么来测试此功能是否按预期工作?您可能还需要验证您的输入事件流是否能够支持此要求。
Does your microservice need to support restarting from the beginning of input stream time, as in the case of full data loss or reprocessing due to a bug? If so, what do you need to know to test if this functionality works as expected? You may also need to validate that your input event streams are capable of supporting this requirement.
您需要哪些数据来确定成功或失败?手动制作的事件数据是否足够?以编程方式创建?是否需要是真实的生产数据?如果有的话,多少钱?
What data do you need to determine success or failure? Is manually crafted event data sufficient? Programmatically created? Does it need to be real production data? If so, how much?
您是否有任何需要测试的性能、负载、吞吐量或扩展问题?
Do you have any performance, load, throughput, or scaling concerns that need to be tested?
您将如何确保您构建的每个微服务不需要完整的本地集成测试解决方案?
How will you go about ensuring that each microservice you build doesn’t require a full home-grown solution to integration testing?
接下来的部分将帮助您了解一些可用的选项,以便您可以对这些问题制定自己的答案。
The next sections will help you understand some of the options available to you, so you can formulate your own answers to these questions.
本地集成测试允许进行大量的功能和非功能测试。这种形式的测试使用将部署微服务的生产环境的本地副本。这至少意味着创建事件代理、模式注册表、任何特定于微服务的数据存储、微服务本身以及任何所需的处理框架,例如当您使用重量级框架或 FaaS 时。你还可以引入容器化、日志记录,甚至容器管理系统,但它们与微服务的业务逻辑并不严格相关,因此并不是绝对必要的。
Local integration testing allows for a significant range of functional and nonfunctional testing. This form of testing uses a local copy of the production environment where the microservice will be deployed. At a minimum this means creating an event broker, schema registry, any microservice-specific data stores, the microservice itself, and any required processing framework, such as when you are using a heavyweight framework or FaaS. You could also introduce containerization, logging, and even the container management system, but they are not strictly related to the business logic of the microservice and so are not absolutely necessary.
建立自己的本地可控环境的最大好处是您可以独立控制每个系统。您可以通过编程方式创建复制实际生产情况的场景,例如间歇性故障、无序事件和网络访问丢失。您还可以测试框架与业务逻辑的集成。本地集成测试还提供了测试水平扩展基本功能的方法,特别是在涉及共同分区和状态的情况下。
The biggest benefit of spinning up your own locally controllable environment is that you get to control each system independently. You can programmatically create scenarios that replicate actual production situations, such as intermittent failures, out-of-order events, and loss of network access. You also get to test the integration of the framework with your business logic. Local integration testing also provides the means to test the basic functionality of horizontal scaling, particularly where copartitioning and state are concerned.
本地集成测试的另一个显着好处是,您可以在同一工作流程中同时有效地测试事件驱动逻辑和请求响应逻辑。您可以完全控制事件何时注入输入流,并且可以在处理事件之前、期间或之后的任何时刻发出请求。将请求-响应 API 视为测试微服务的另一个事件源可能会有所帮助。
Another significant benefit of local integration testing is that you can effectively test both event-driven and request-response logic at the same time, in the same workflows. You have full control over when events are injected into the input streams, and can issue requests at any point before, during, or after the events have been processed. It may be helpful to think of the request-response API as just another source of events for the purposes of testing your microservice.
让我们看一下每个系统组件提供的一些选项。
Let’s take a look at some of the options provided by each system component.
创建和删除事件流
对输入流应用选择性事件排序,以执行基于时间的逻辑、无序事件和上游生产者故障
修改分区计数
诱导代理故障和恢复
引发事件流可用性故障和恢复
Create and delete event streams
Apply selective event ordering for input streams to exercise time-based logic, out-of-order events, and upstream producer failures
Modify partition counts
Induce broker failures and recovery
Induce event stream availability failures and recovery
为给定的事件流发布进化兼容的模式并使用它们来生成输入事件
引发故障和恢复
Publish evolutionary-compatible schemas for a given event stream and use them to produce input events
Induce failures and recovery
对现有表进行架构更改(如果适用)
更改存储过程(如果适用)
修改应用程序实例计数时重建内部状态(如果适用)
引发故障和恢复
Make schema changes to existing tables (if applicable)
Make changes to stored procedures (if applicable)
Rebuild internal state (if applicable) when the application instance count is modified
Induce failures and recovery
应用程序和处理框架通常是相互交织的,您可能需要提供完整的框架实现来进行测试,就像 FaaS 和重量级框架解决方案的情况一样。该框架提供的功能包括:
通过内部事件流(轻量级)或洗牌机制(重量级)进行洗牌,以确保正确的共同分区和数据局部性
检查点、故障以及从检查点恢复
诱导工作实例失败以模拟丢失应用程序实例(重量级框架)
The application and the processing framework are typically intertwined, and you may need to provide a full-framework implementation for testing, as in the case of FaaS and heavyweight framework solutions. The framework provides functionality such as:
Shuffling via internal event streams (lightweight) or shuffle mechanism (heavyweight) to ensure proper copartitioning and data locality
Checkpointing, failures, and recovery from checkpoints
Inducing a worker instance failure to mimic losing an application instance (heavyweight frameworks)
应用程序级控制主要涉及管理在任何给定时间运行的实例数量。集成测试应包括扩展实例计数(动态,如果支持)以确保:
重新平衡如期发生
从检查点或变更日志流恢复内部状态,并保留数据局部性
外部状态访问不受影响
对状态数据的请求-响应访问不受应用程序实例计数变化的影响
Application-level control predominantly involves managing the number of instances running at any given time. Integration testing should include scaling the instance count (dynamically, if supported) to ensure that:
Rebalancing occurs as expected
Internal state is restored from checkpoints or changelog streams, and data locality is preserved
External state access is unaffected
Request-response access to the stateful data is unaffected by changes in application instance count
完全控制所有这些系统的目的是确保您的微服务在各种故障模式、不利条件和不同的处理能力下仍能按预期工作。
The point of having full control over all of these systems is to ensure that your microservice will still work as intended through various failure modes, adverse conditions, and with varying processing capacity.
执行本地集成测试有两种主要方法。第一个涉及嵌入可以严格存在于代码中的测试库。这些并不适用于所有微服务解决方案,并且往往严重依赖于语言和框架支持。第二个选项涉及创建一个本地环境,其中安装了每个必要的组件,并且可以根据需要进行控制。我们将首先了解这些,然后研究用于测试依赖于托管服务的微服务的选项。
There are two main ways to perform local integration tests. The first involves embedding testing libraries that can live strictly in your code. These are not available for all microservice solutions and tend to depend heavily on both language and framework support. The second option involves creating a local environment where each of the necessary components is installed and can be controlled as required. We’ll take a look at these first, then follow up by investigating options for testing microservices that rely on hosted services.
将测试库嵌入到代码中是迄今为止最窄的选项,尽管它可以很好地工作,具体取决于您的客户端、代理和框架编程语言兼容性的排列方式。在这种方法中,测试代码在与应用程序相同的可执行文件中启动必要的组件。
Embedding testing libraries into your code is by far the narrowest of the options, though it can work very well depending on how your client, broker, and framework programming language compatibilities line up. In this approach, the test code starts the necessary components within the same executable as the application.
例如,Kafka Streams 应用程序的测试代码启动其自己的 Kafka 代理、模式注册表和微服务拓扑实例。然后,测试代码可以启动和停止拓扑实例、发布事件、等待响应、导致代理中断以及引发其他故障模式。终止后,所有组件都将终止,并且状态将被清除。考虑以下伪代码(为简洁起见,跳过声明和实例化):
For example, the test code of a Kafka Streams application starts its own Kafka broker, schema registry, and microservice topology instances. The test code can then start and stop topology instances, publish events, await responses, incur broker outages, and induce other failure modes. Upon termination, all of the components are terminated, and the state is cleaned up. Consider the following pseudocode (declarations and instantiations skipped for brevity):
broker.start(brokerUrl,brokerPort,...);schemaRegistry.start(schemaRegistryUrl,srPort,...);//The first instance of the microservicetopologyOne.start(brokerUrl,schemaRegistryUrl,inputStreamOne,inputStreamTwo...);//A second instance of the same microservicetopologyTwo.start(brokerUrl,schemaRegistryUrl,inputStream,inputStreamTwo,...);//Publish some test data to input stream 1producer.publish(inputStreamOne,...);//Publish some test data to input stream 2producer.publish(inputStreamTwo,...);//Wait a short amount of time. Not the best way to do it, but you get the ideaThread.sleep(5000);//Now mimic topologyOne failingtopologyOne.stop();//Check the output of the output topic. Is it as expected?event=consumer.consume(outputTopic,...)//Shut down the remaining components if no more testing is to be donetopologyTwo.stop()schemaRegistry.stop()broker.stop()if(event...)//validate the consumer output//pass the test if correctelse//fail the test.
broker.start(brokerUrl,brokerPort,...);schemaRegistry.start(schemaRegistryUrl,srPort,...);//The first instance of the microservicetopologyOne.start(brokerUrl,schemaRegistryUrl,inputStreamOne,inputStreamTwo...);//A second instance of the same microservicetopologyTwo.start(brokerUrl,schemaRegistryUrl,inputStream,inputStreamTwo,...);//Publish some test data to input stream 1producer.publish(inputStreamOne,...);//Publish some test data to input stream 2producer.publish(inputStreamTwo,...);//Wait a short amount of time. Not the best way to do it, but you get the ideaThread.sleep(5000);//Now mimic topologyOne failingtopologyOne.stop();//Check the output of the output topic. Is it as expected?event=consumer.consume(outputTopic,...)//Shut down the remaining components if no more testing is to be donetopologyTwo.stop()schemaRegistry.stop()broker.stop()if(event...)//validate the consumer output//pass the test if correctelse//fail the test.
Kafka Streams 是一个特别相关的示例,因为它说明了这种方法的局限性。应用程序代码、代理和 Confluence 模式注册表都是基于 JVM 的,因此您需要一个基于 JVM 的应用程序来以编程方式控制同一运行时中的所有内容。其他开源重量级框架也可以工作,尽管需要一些额外的开销来处理创建主实例和工作实例。请记住,由于这些重量级框架也几乎普遍基于 JVM,因此在撰写本文时,该策略主要是仅使用 JVM 的方法。虽然当然可以使用变通方法以这种方式测试非基于 JVM 的应用程序,但该过程远没有这么简单。
Kafka Streams is a particularly relevant example because it illustrates the limited nature of this approach. The application code, the broker, and the Confluent schema registry are all JVM-based, so you need a JVM-based application to programmatically control everything within the same runtime. Other open source heavyweight frameworks may also work, though some extra overhead is required to handle creating both the master instance and the worker instance. Keep in mind that because these heavyweight frameworks are also almost universally JVM-based, this strategy is predominantly a JVM-only approach at the time of this writing. While it is certainly possible to use workarounds to test non-JVM-based applications in this manner, that process is not nearly as simple.
设置执行这些测试的环境的一种选择是在本地安装和配置所有必需的系统。这是一种低开销的方法,特别是如果您刚刚开始使用微服务,但如果每个团队成员都必须这样做,那么如果他们每个人都运行略有不同的版本,那么调试就会变得昂贵且复杂。与大多数微服务一样,通常最好避免重复步骤,而是提供可消除开销的支持工具。
One option for setting up an environment to perform these tests is simply to install and configure all required systems locally. This is a low-overhead approach particularly if you’re just starting out with microservices, but if every teammate must do the same, it becomes expensive and complex to debug if they’re each running slightly different versions. As with most things microservices, it’s often best to avoid duplicating steps and instead provide supportive tooling that eliminates the overhead.
更灵活的选择是创建一个安装和配置所有必需组件的容器。任何想要以这种方式测试其应用程序的团队都可以使用该容器。您可以维护开源贡献模型(即使是组织内部的),允许添加修复、更新和新 功能,以造福所有团队。该模型足够灵活,可以与任何编程语言一起使用,但与允许与容器中的系统组件轻松通信的编程 API 一起使用要容易得多。轻量级处理框架示例如图15-1所示,以及在容器内部创建的模式注册表、事件代理和必要的主题。微服务实例本身在容器外部执行,并且仅从其测试配置文件引用代理和架构注册表的地址。
A more flexible option is to create a single container that has all of the necessary components installed and configured. This container can be used by any team that wants to test its application in this way. You can maintain an open source contribution model (even if internal to the organization), allowing fixes, updates, and new features to be added back for the benefit of all teams. This model is flexible enough to be used with any programming language, though it’s far easier to use with a programmatic API that allows for easy communication with the system components in the container. A lightweight processing framework example is shown in Figure 15-1, with the schema registry, event broker, and necessary topics created internally to the container. The microservice instance itself is executed externally to the container, and simply references the addresses of the broker and schema registry from its testing config file.
本地集成测试环境可能还需要提供托管服务,例如托管事件代理、重量级框架或 FaaS 平台。虽然某些托管服务可能具有您可以运行的开源选项(例如开源 Kafka 而不是托管 Kafka),但并非所有托管服务都有这些替代方案。例如,微软的事件中心、谷歌的 PubSub 和亚马逊的 Kinesis 都是专有且封闭的,完整的实现无法下载。在这种情况下,您能做的最好的事情就是使用这些公司或开源项目提供的任何模拟器、库或组件。
A local integration testing environment may also need to provide hosted services, such as a hosted event broker, heavyweight framework, or FaaS platform. While some hosted services may have open source options you can run instead (such as open source Kafka instead of hosted Kafka), not all hosted services have these alternatives. For example, Microsoft’s Event Hubs, Google’s PubSub, and Amazon’s Kinesis are all proprietary and closed, with full implementations unavailable for download. In this situation, the best you can do is use whatever emulators, libraries, or components are available from these companies or open source initiatives.
例如,Google 的 PubSub 有一个模拟器,可以提供足够的本地测试功能,LocalStack 提供的 Kinesis 开源版本(以及许多其他 Amazon 服务)也是如此。不幸的是,Microsoft Azure 的事件中心当前没有模拟器,开源世界中也没有可用的实现。不过,Azure 事件中心客户端确实允许您使用 Apache Kafka 来代替它,尽管并非所有功能都受支持。
Google’s PubSub, for example, has an emulator that can provide sufficient local testing functionality, as does an open source version of Kinesis (and many other Amazon services) provided by LocalStack. Unfortunately, Microsoft Azure’s Event Hubs does not currently have an emulator, nor is an implementation of it available in the open source world. Azure Event Hub clients do, however, allow you to use Apache Kafka in its place, though not all features are supported.
使用 FaaS 平台的应用程序可以利用托管服务提供的本地测试库。Google Cloud 函数可以在本地进行测试, Amazon 的 Lambda 函数和Microsoft Azure 的函数也可以。第 9 章中讨论的开源解决方案 OpenWhisk、OpenFaaS 和 Kubeless提供了类似的测试机制,您可以通过快速网络搜索找到这些机制。这些选项允许您在本地配置完整的 FaaS 环境,以便您可以在配置为尽可能类似于生产的平台上进行测试。
Applications using FaaS platforms can leverage local testing libraries provided by the hosting service. Google Cloud functions can be tested locally, as can Amazon’s Lambda functions and Microsoft Azure’s functions. The open source solutions OpenWhisk, OpenFaaS, and Kubeless, as discussed in Chapter 9, provide similar testing mechanisms, which you can find via a quick web search. These options allow you to configure a complete FaaS environment locally, such that you can test on a platform configured to be as similar to production as possible.
使用重量级框架为应用程序建立集成测试环境与为 FaaS 框架建立集成测试环境的过程类似。每个都需要安装和配置框架,应用程序将处理作业直接提交给框架。对于重量级框架,典型的单容器安装只需要并行运行主实例和工作实例以及事件代理和任何其他依赖项。设置重量级框架后,您只需将处理作业提交给主服务器并等待输出事件流上的测试输出。图 15-2中展示了一个示例,其中整套依赖项已被容器化,以便于在开发人员之间分发。
Establishing an integration testing environment for applications using heavyweight frameworks is similar to the process of establishing one for FaaS frameworks. Each requires that the framework be installed and configured, with the application submitting the processing job directly to the framework. With heavyweight frameworks, a typical single-container installation will just need to run the master and worker instances side-by-side along with the event broker and any other dependencies. With the heavyweight framework set up, you simply need to submit the processing job to the master and await test output on the output event streams. An example is illustrated in Figure 15-2, where the entire set of dependencies has been containerized for easy distribution among developers.
生产中使用的某些服务可能根本没有任何本地可用选项,这对于开发和集成测试来说都是一个缺点。当前的一个例子是 Microsoft Azure 事件中心没有任何模拟器。缺乏本地可用的实现意味着除了这些应用程序的集成测试环境之外,还必须为每个开发人员配置远程环境。这也是界限开始变得模糊的地方,因为迄今为止的集成测试主要是在一次性、易于管理的本地环境中隔离单个应用程序实例。这种情况下产生的开销可能是独立开发和集成测试工作的真正障碍,因此在继续之前一定要仔细考虑。
Some services used in production may simply not have any locally available options, and this is a disadvantage for both development and integration testing. A current example is the absence of any emulator for Microsoft Azure’s Event Hub. The lack of a locally available implementation means that remote environments must be provisioned for each developer, in addition to integration testing environments for these applications. This is also where lines can begin to blur, as integration testing up to this point has been primarily about isolating a single application instance in a disposable, easily managed, local environment. The overhead incurred in this scenario can be a real impediment to independent development and integration testing efforts, so be sure to give it careful consideration before moving forward.
缓解这个问题通常需要与基础设施团队密切协调,以确保可以通过访问控制独立配置独立的测试环境,或者可以创建一个大型的公共环境供所有人使用(这有其自身的问题,如稍后讨论的)章节)。开发人员必须将本地暂存环境连接到远程资源,可能会出现安全问题。远程暂存环境的清理和管理也可能成为问题。应对这一挑战的方法有很多,但这种情况可能带来的问题太大,无法在这里全面解决。
Alleviating this issue generally requires close coordination with infrastructure teams to ensure either that independent testing environments can be independently provisioned via access controls or that a large, common environment can be created for all to use (this has its own issues, as discussed later in the chapter). Security issues may arise from developers having to connect their local staging environment to remote resources. Cleanup and management of the remote staging environment(s) can also become problematic. There are many ways to approach this challenge, but the problems that the situation may pose are too large to comprehensively tackle here.
好消息是,大多数最大的闭源服务提供商正在努力提供本地开发和测试选项,因此先行者迟早都会拥有这些可用的。同时,请谨慎选择服务,并考虑是否可以使用本地开发和集成测试选项。
The good news is that most of the biggest closed source service providers are making strong efforts to provide local options for development and testing, so in time the forerunners will all have these available. In the meantime, be cautious about your selection of services and consider whether a local option for development and integration testing is available.
完整的远程集成测试使您能够执行在本地环境中难以执行的特定测试。例如,性能和负载测试对于确保被测微服务实现其服务级别目标至关重要。事件处理吞吐量、请求响应延迟、实例扩展和故障恢复都是通过完整的集成测试实现的。
Full remote integration testing enables you to perform specific tests that are difficult to conduct in local environments. For example, performance and load testing are essential for ensuring that the microservice under test achieves its service-level objectives. Event processing throughput, request-response latency, instance scaling, and failure recovery are all enabled by full integration testing.
完整集成测试的目标是创建一个尽可能接近生产环境的环境,包括运行应用程序的事件流、事件数据量、事件模式和请求响应模式(如果适用)。
The goal of full integration testing is to create an environment as close to possible as that of production, including event streams, event data volume, event schemas, and request-response patterns (if applicable), in which to run the application.
完整的集成测试通常通过以下三种方式之一完成。您可以使用临时集成环境,并在测试完成后将其丢弃。您可以使用通用测试环境,该环境在集成测试之间持续存在并由多个团队使用。最后,您可以使用生产环境本身进行测试。
Full integration testing is generally done in one of three following ways. You can use a temporary integration environment and discard it once testing is complete. You can use a common testing environment, which persists between integration tests and is used by multiple teams. Finally, you can use the production environment itself for testing.
“集群创建和管理”研究了以编程方式生成事件代理和计算资源管理器的优势。您可以利用这些工具为集成测试生成临时环境。可以创建一组单独的代理以及单独保留的计算资源来运行测试中的容器化微服务。使用这种方法进行全面集成测试的另一个好处是,它可以定期执行创建新代理和计算环境的过程。这确保了脚本中发生的任何损坏或配置中的任何错误都将在下一次集成测试中暴露出来。
“Cluster Creation and Management” examined the advantages of having programmatically generated event brokers and compute resource managers. You can leverage these tools to generate temporary environments for your integration testing. A separate set of brokers can be created along with individually reserved compute resources to run the containerized microservices under test. One added benefit of using this approach for full integration testing is that it regularly exercises the process of creating new brokers and compute environments. This ensures that any breakages that occur in the scripts or any bugs in the configuration will be exposed at the next integration test.
新环境中的下一个问题是它缺乏事件流和事件数据。当然,这些对于测试微服务都是必不可少的。您可以通过直接询问用户或使用工具访问的微服务代码库中的配置文件来获取要创建的事件流的名称。分区计数必须反映生产系统的分区计数,以确保正确执行微服务的扩展、共同分区和重新分区逻辑。
The next issue in a newly brought-up environment is that it lacks both event streams and event data. These are, of course, both essential for testing your microservice. You can obtain the names of the event streams to create either by directly asking the user or by using a configuration file within the microservice codebase to be accessed by the tooling. The partition count must mirror that of the production system to ensure that the microservice’s scaling, copartitioning, and repartitioning logic is correctly exercised.
生成事件流后,下一步就是用事件填充它们。这可以使用生产数据、专门策划的测试数据集或临时的、以编程方式生成的数据来完成。
Once the event streams have been generated, the next step is to populate them with events. This can be done using production data, specially curated testing data sets, or ad hoc, programmatically generated data.
可以将事件从生产集群复制到测试集群中新创建的事件流。这就是“跨集群事件数据复制”中描述的复制工具发挥作用的地方,因为该工具可用于从生产环境复制特定事件流并加载事件。您必须考虑可能阻止生产获取数据的任何安全和访问限制。
Events can be copied from the production cluster over to the newly created event streams in the testing cluster. This is where the replication tooling described in “Cross-Cluster Event Data Replication” comes into play, as this same tooling can be used to replicate specific event streams from production and load the events. You must account for any security and access restrictions that may prevent production from obtaining the data.
它准确地反映了生产数据。
您可以根据需要复制任意数量的事件。
完全隔离的环境可以防止其他被测微服务无意中影响您的测试。
It accurately reflects production data.
You can copy as many or as few events as required.
The fully isolated environment prevents other microservices under test from inadvertently affecting your testing.
除非您充分规划并建立了代理配额,否则复制数据可能会影响生产性能。
它可能需要复制大量数据,特别是对于长期存在的实体。
您必须考虑包含敏感信息的事件流。
它需要大量投资来简化创建和复制过程,以减少使用障碍。
它可能会暴露敏感的生产事件。
Copying data may affect production performance unless you have adequately planned and established broker quotas.
It may require copying substantial amounts of data, especially in the case of long-lived entities.
You must account for event streams containing sensitive information.
It requires significant investment in streamlining the creation and copying process to reduce barriers to usage.
It may expose sensitive production events.
策划事件允许您在集成测试中使用具有特定属性、值以及与其他事件的关系的事件。这些事件需要存储在稳定且安全的地方,这样它们就不会被意外或无意地覆盖、损坏或丢失。此策略通常用于单个共享测试环境(稍后详细介绍),但您也可以通过将事件从持久数据存储加载到用户指定的流中来应用它,类似于从生产中复制事件。
Curated events allow you to use events with specific properties, values, and relationships to other events in integration testing. These events need to be stored somewhere stable and secure, where they can’t be accidentally or inadvertently overwritten, corrupted, or lost. This strategy is often used in single shared testing environments (more on this later), but you can also apply it by loading the events out of a durable data store into the user-specified streams, similar to copying events from production.
它涉及较小的数据集。
它经过精心策划,以确保特定的价值观和关系。
对生产没有影响。
It involves a smaller set of data.
It’s carefully curated to ensure specific values and relationships.
It has no impact on production.
维护的开销很大。
数据可能会变得陈旧。
必须处理新的事件流。
必须处理架构更改。
较少使用的事件流可能不可用。
There’s significant overhead to maintain.
Data can become stale.
New event streams must be handled.
Schema changes must be handled.
Lesser-used event streams may not be available.
虽然许多缺点可以通过严格的操作流程来缓解,但这种策略最终往往会遵循许多组织的文档记录相同的路径。它的初衷是好的,但仍然过时,不一定相关,而且似乎总是比其他工作优先级低。
While many of these disadvantages can be mitigated through strict operational processes, this strategy often ends up following the same pathway that documentation tends to at many organizations. It is well intentioned but remains out-of-date, not necessarily relevant, and seemingly always a lower priority than other work.
以编程方式创建模拟事件是填充事件流的另一种可能性。您可以从架构注册表获取架构并生成遵循此架构定义的事件。您甚至可以采用该模式的旧版本并为这些版本生成事件。
Programmatic creation of mock events is another possibility for populating event streams. You can obtain the schema from the schema registry and generate events that adhere to this schema definition. You can even take older versions of that schema and generate events for those too.
这种方法的复杂性来自于确保事件与其他事件具有正确的关系,特别是如果任何服务正在流之间进行连接或跨不同类型的事件进行聚合。将多个事件连接在一起的微服务需要创建具有匹配主/外键的事件才能正确执行服务的连接逻辑。虽然这通常不是一个重大问题(特别是因为微服务代码表达了业务逻辑所需的关系),但它确实将其留给了该数据的创建者来确保其范围正确并且所有数据都落入预期范围和价值观。
The complexity of this approach comes from ensuring that there are events with the proper relationships to other events, particularly if any of the services are doing joins between streams or aggregations across different types of events. A microservice that joins multiple events together will require that events with matching primary/foreign keys have been created to properly exercise the joining logic of the service. While this is not typically a significant issue (especially since the microservice code expresses which relationships the business logic requires), it does leave it up to the creators of this data to ensure that it is properly scoped and that all data falls into the expected ranges and values.
它不需要生产集群提供任何数据,并且不会对生产性能产生负面影响。
您可以使用模糊测试工具来创建事件数据、测试边界条件以及其他潜在的格式错误和半格式字段。
您可以创建生产数据中不可用的特定测试用例,确保覆盖极端情况。
它允许您利用第三方工具以编程方式创建测试数据(例如Confluence Avro 工具)。
It doesn’t require the production cluster to provide any data and can’t negatively affect production performance.
You can use fuzzing tools to create event data, testing boundary conditions, and other potential malformed and semiformed fields.
You can create specific test cases that aren’t available in production data, ensuring corner cases are covered.
It allows you to leverage third-party tools for programmatically creating testing data (for example, Confluent Avro tools).
It requires much more attention to creating realistic data than other options.
The created data is still not fully accurate when compared to the production distribution. For example, production data may have a serious disparity in data volume due to key distribution that doesn’t show up in mock data.
The created data may inaccurately represent certain fields. For example, parsing a string field in a particular way for business operations may pass with the created test data, but fail in a subset of production data.
另一种选择是创建一个单一的测试环境,其中包含所有驻留在同一事件代理中的事件流共享池。这些流由代表生产数据子集的测试数据或如前所述精心设计的测试数据填充。此选项提供了低开销的测试环境,但将事件流和数据的管理工作转移给了应用程序开发人员。
Another option involves creating a single testing environment with a shared pool of event streams all residing within the same event broker. These streams are populated by testing data that represents a subset of production data, or carefully crafted testing data as previously discussed. This option provides a low-overhead a testing environment, but offloads the management of the event streams and the data onto the application developers.
很容易上手。
您只需维护一个测试环境的基础架构。
它与生产工作负载隔离。
It’s easy to get started.
You only need to maintain infrastructure for one testing environment.
It is isolated from production workloads.
它遭受了“公地悲剧”。碎片化和废弃的事件流可能会导致很难区分哪些流对于测试输入有效,哪些只是之前未清理的测试的输出。
被测系统不一定是孤立的。例如,同时运行大规模性能测试的服务可能会影响彼此的结果。
可能会产生与其他服务的输入事件流不兼容的事件。
事件流数据不可避免地会变得陈旧,必须使用新事件进行更新。
它不准确地代表了生产中发现的整个事件范围。
It is subject to the “tragedy of the commons.” Fragmented and abandoned event streams can make it difficult to distinguish which streams are valid for testing input and which are simply the output of previous tests that were not cleaned up.
Systems under test are not necessarily isolated. For example, services running simultaneous large-scale performance testing can affect each other’s results.
Incompatible events may be produced to other services’ input event streams.
Event stream data inevitably becomes stale and must be updated with newer events.
It inaccurately represents the entire range of events found in production.
就可用性而言,此策略是最差的选项,因为事件代理最终会成为混乱事件流和损坏数据的垃圾场。
This strategy is the worst of the options in terms of usability, as the event broker eventually becomes a dumping ground of confusing event streams and broken data.
与其他应用程序测试的隔离很难实现,特别是因为一个微服务的输出流通常是另一服务的输入。仔细管理数据流、严格的命名约定以及对写入事件流的限制可以帮助减轻这些缺点,但环境维护者和用户需要勤奋和遵守纪律。
Isolation from other application testing is difficult to achieve, particularly as the output stream of one microservice is usually the input to another service. Careful curation of data streams, strict naming conventions, and restrictions to writing to event streams can help mitigate the disadvantages, but environment maintainers and users will need to exercise diligence and discipline.
您还可以在生产环境中测试微服务(注意:要小心)。微服务可以启动、从其输入事件流中消费、应用业务逻辑并生成输出。最常见的方法是让微服务使用自己指定的输出事件流和状态存储,这样就不会影响现有的生产系统。当同一微服务的先前版本与正在测试的新版本一起运行时,这一点尤其重要。
You can also test microservices in the production environment (note: be careful). The microservice can be spun up, consume from its input event streams, apply business logic, and produce output. The most common approach is to have the microservice use its own designated output event streams and state stores, such that it doesn’t affect the existing production systems. This is particularly important when a previous version of the same microservice is running alongside the new one under test.
您拥有制作活动的完全访问权限。
它利用生产安全模型来确保遵循正确的访问 协议。
它非常适合对应用程序进行冒烟测试。
您不需要维护单独的测试环境。
You have complete access to production events.
It leverages production security models to ensure proper access protocols are followed.
It is excellent for smoke-testing an application.
You do not need to maintain a separate testing environment.
存在影响生产能力的风险,特别是在工作量很大的情况下。它不适合负载和性能测试。
您必须仔细清理测试期间创建的所有资源,例如事件流、使用者组、访问控制权限和状态存储。这与通用暂存环境选项的要求类似。
它需要工具支持来将测试中的微服务和事件流与“真正的生产”微服务分开,特别是当您进行长时间测试时。这包括用于管理和部署微服务的资源,因为生产环境的每个观察者都必须能够识别哪些服务是真正的生产服务,哪些是正在测试的服务。
There’s a risk of affecting production capacity, especially if workload is high. It’s not suitable for load and performance testing.
You must carefully clean up any resources created during testing, such as event streams, consumer groups, access control permissions, and state stores. This is similar to the requirements for the common staging environment option.
It requires tooling support to keep microservices and event streams under test separate from “true production” microservices, particularly when you are testing over a long period of time. This includes the resources used to manage and deploy microservices, as each observer to the production environment must be able to identify which services are the true production ones and which are those under test.
微服务模块化的好处是您不必只选择一种方法来执行测试。您可以根据需要使用任何选项,为其他项目切换到不同的选项,并根据需求的变化更新您的测试方法。对多集群事件代理和事件复制功能的支持工具的投资将在很大程度上决定您的测试选项。
The nice thing about the modularity of microservices is that you don’t have to choose just one way to perform your tests. You can use any option as needed switch to a different one for other projects, and update your testing methodology as requirements change. Investments in supportive tooling for multicluster event brokers and event copying capabilities will largely determine your testing options.
如果您几乎没有支持工具,那么您很可能最终会得到一个共享的测试事件代理,其中包含由不同团队和系统生成的事件流的大杂烩。您可能会看到可用于测试的“良好”事件流的混合体,以及带有“-testing-01”、“-testing-02”、“-testing-02-final”等后缀的事件流“-testing-02-final-v2。” 事件数据可能可靠、最新,或者架构格式正确,也可能不可靠。部落知识在这个世界中发挥着重要作用,并且很难确保您的测试充分反映您的服务的生产环境。此外,连续可用的临时集群的成本要高得多,该集群还必须启用性能测试、加载大量数据并提供无限期的事件存储。
If you have little to no supportive tooling, you’re most likely going to end up with a single, shared testing event broker with a hodge-podge of event streams generated by various teams and systems. You’ll likely see a mixture of “good” event streams that you can use for testing, and event streams with suffixes like “-testing-01,” “-testing-02,” “-testing-02-final,” and “-testing-02-final-v2.” Event data may or may not be reliable, up-to-date, or in the correct schema format. Tribal knowledge plays a large role in this world, and it can be difficult to ensure your testing sufficiently reflects your service’s production environment. In addition, costs are much higher for a continuously available staging cluster that must also enable performance testing, load large amounts of data, and provide event stores with indefinite duration.
通过对工具的适当投资,每个微服务都可以启动自己的专用集群,用事件流填充它,将一些生产数据复制到其中,并在几乎相同的生产环境中运行测试。一旦测试完成,就可以拆除集群,从而消除否则会留在共享集群中的测试工件。达到这一阶段的开销是巨大的,但投资可以释放多集群工作、冗余和灾难恢复选项,而这些选项很难通过其他方式获得(更多详细信息,请参阅第 14 章)。
With proper investment in tooling, each microservice can bring up its own dedicated cluster, populate it with event streams, copy some production data into it, and run the test in a nearly identical production environment. The cluster can be torn down once testing is completed, eliminating testing artifacts that would otherwise stick around in a shared cluster. The overhead for getting to this stage is significant, but the investment unlocks multicluster efforts, redundancy, and disaster recovery options that are difficult to obtain otherwise (see Chapter 14 for more details).
这并不是说单个共享测试集群本质上是不好的。努力标记干净可靠的源流很重要,删除未使用的测试工件也很重要。具体的、条文化的职责可以确保对暂存事件数据可靠性的期望由拥有生产事件数据的团队管理。团队还必须协调性能和负载测试,以确保它们不会影响彼此的结果。随着您的团队改进其多集群和事件复制工具,团队可以开始迁移到自己动态创建的测试集群。
This isn’t to say a single shared testing cluster is inherently bad. Diligence in marking clean and reliable source streams is important, as is deleting unused testing artifacts. Specific, codified responsibilities can ensure that expectations around staging event-data reliability are managed by the teams that own the production event data. Teams must also coordinate performance and load testing to ensure they do not affect each other’s results. As your team improves its multicluster and event copying tooling, teams can begin to migrate to their own dynamically created testing clusters.
事件驱动的微服务主要从事件流获取输入数据。您可以通过多种方式创建和填充这些流,包括从生产中复制数据、整理特定数据集以及根据架构自动生成事件。每种方法都有其自身的优点和缺点,但它们都依赖于支持工具来创建、填充和管理这些事件流。
Event-driven microservices predominantly source their input data from event streams. You can create and populate these streams in a variety of ways, including copying data from production, curating specific data sets, and automatically generating events based on the schema. Each method has its own advantages and disadvantages, but all of them rely on supportive tooling to create, populate, and manage these event streams.
建立一个测试微服务的环境应该是一项协作工作。您组织中的其他开发人员和工程师无疑将从通用测试平台中受益,因此您应该考虑投资工具来简化您的测试流程。以编程方式启动环境(包括事件流的数量)可以显着减少为每个测试中的微服务设置环境的开销。
Establishing an environment in which to test your microservice should be a collaborative effort. Other developers and engineers in your organization will undoubtedly benefit from a common testing platform, so you should consider investing in tooling to streamline your testing processes. Programmatic bringup of environments, including the population of event streams, can significantly reduce the overhead of setting up environments for each microservice under test.
当工具投资较低时,单一共享测试环境是一种常见策略。代价是管理事件数据、确保有效性和澄清所有权方面的难度增加。一次性环境是更好的选择,因为它们为测试中的服务提供隔离,并减少由多个租赁问题引起的风险和缺点。这些选项往往需要对通用支持工具进行更多投资,但从长远来看可以节省大量时间和精力。作为一个额外的好处,使用编程环境启动和事件复制工具可以更好地为您的组织做好灾难恢复的准备。
A single shared testing environment is a common strategy to employ when investment in tooling is low. The tradeoff is the increased difficulty in managing the event data, ensuring validity, and clarifying ownership. Disposable environments are a preferable alternative, as they provide isolation for services under test and reduce the risks and shortcomings caused by multiple tenancy issues. These options tend to require more investment in common supportive tooling, but save significant time and effort in the long run. As an added benefit, using programmatic environment bringup and event copying tooling can better prepare your organization for disaster recovery.
部署事件驱动的微服务可能具有挑战性。随着组织内微服务数量的增加,标准化部署流程的重要性也随之增加。仅管理几十个服务的组织可以摆脱一些自定义部署流程,但任何认真投资微服务、事件驱动或其他方式的组织都必须投资于标准化和简化其部署流程。
Deploying event-driven microservices can be challenging. As the number of microservices within an organization increases, so does the importance of having standardized deployment processes in place. An organization managing only a few dozen services can get away with a few custom deployment processes, but any organization seriously invested in microservices, event-driven or otherwise, must invest in standardization and streamlining its deployment processes.
There are a number of principles that drive deployment processes:
团队应该控制自己的测试和部署过程,并拥有自行决定部署微服务的自主权。
Teams should control their own testing and deployment process and have the autonomy to deploy their microservices at their discretion.
服务之间的部署过程应该保持一致。应使用已有的部署流程创建新的微服务。正如稍后讨论的,这通常是通过持续集成框架来完成的。
The deployment process should be consistent between services. A new microservice should be created with a deployment process already available to it. This is commonly accomplished with a continuous integration framework, as is discussed shortly.
部署可能需要团队重置消费者组偏移、清除状态存储、检查和更新模式演变以及删除内部事件流。支持工具提供这些功能,以实现部署的进一步自动化和支持团队自治。
Deployments may require teams to reset consumer group offsets, purge state stores, check and update schema evolution, and delete internal event streams. Supportive tooling provides these functions to enable further automation of deployment and support team autonomy.
重新使用输入事件流可能非常耗时,导致下游消费者的结果过时。此外,该微服务随后可能会生成大量输出事件,导致下游消费者再次出现高负载。非常大的事件流和拥有大量消费者的事件流可能会出现处理能力需求的显着激增。您还必须考虑副作用,特别是那些可能对客户造成干扰的副作用(例如,重新发送多年的促销电子邮件)。
Reconsuming input event streams can be time-consuming, leading to stale results for downstream consumers. Additionally, this microservice may subsequently generate a large volume of output events, causing another high load for downstream consumers. Very large event streams and those with large amounts of consumers may see nontrivial surges in processing power requirements. You must also consider side effects, particularly those that can be disruptive to customers (e.g., resending multiple years’ worth of promotional emails).
部署可能会破坏其他服务。例如,重建状态存储可能会导致大量停机,而重新处理输入事件流可能会生成大量事件。确保在部署过程中遵守所有 SLA。
Deployments may be disruptive to other services. For instance, rebuilding state stores can result in a significant amount of downtime, while reprocessing input event streams may generate a significant number of events. Ensure that all SLAs are honored during the deployment process.
部署可能要求其他服务更改其 API 或数据模型,例如在与 REST API 交互或引入域架构更改时。应尽可能减少这些更改,因为它们侵犯了其他团队仅在业务需求变化时部署其服务的自主权。
Deployments may require that other services change their APIs or data models, such as when interacting with a REST API or introducing a domain schema change. These changes should be minimized whenever possible, as they violate the other team’s autonomy for deploying their services only when required by shifting business requirements.
在某些情况下,破坏架构更改可能是不可避免的,需要创建新的事件流并与下游消费者重新协商数据契约。确保在任何部署之前进行这些讨论,并确保消费者的迁移计划到位。
Breaking schema changes may be inevitable in some circumstances, requiring the creation of new event streams and a renegotiation of the data contract with downstream consumers. Ensure that these discussions happen before any deployment and that a migration plan for consumers is in place.
微服务应该是可独立部署的,否则就是一种反模式。如果特定的微服务部署定期要求其他微服务同步其部署,则表明它们的限界上下文定义不明确,应进行审查。
Microservices should be independently deployable, and it is an anti-pattern if they are not. If a particular microservice deployment regularly requires other microservices to synchronize their deployments, it is an indicator that their bounded contexts are ill-defined and should be reviewed.
微服务部署架构有几个主要组件,每个组件都发挥着关键作用。该架构可以大致分为两个主要组件:用于构建和部署代码的系统以及微服务使用的计算资源。
There are several major components of the microservice deployment architecture, each of which plays a pivotal role. This architecture can be roughly broken down into two main components: the system used to build and deploy the code, and the compute resources used by the microservices.
持续集成、交付和部署系统允许在将代码更改提交到存储库时构建、测试和部署微服务。这是您必须支付的微服务税的一部分,才能成功地管理和大规模部署微服务。这些系统允许微服务所有者决定何时部署其微服务,这对于扩大组织中使用的微服务数量至关重要。
Continuous integration, delivery, and deployment systems allow for microservices to be built, tested, and deployed as code changes are committed to the repository. This is part of the microservice tax that you must pay to successfully manage and deploy microservices at scale. These systems allow microservice owners to decide when to deploy their microservices, which is essential for scaling up the number of microservices used in an organization.
持续集成(CI) 是将多个贡献者的代码更改自动集成到单个软件项目中的实践。代码更改由管理微服务的团队自行决定集成,目的是减少代码更改与在生产中部署之间的时间。CI框架允许在代码合并到主分支时自动执行流程,包括构建操作、单元测试和集成测试操作。其他 CI 流程可能包括验证代码风格和执行模式演化验证。准备部署的容器或虚拟机是 CI 管道的最终输出。
Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. Changes from code are integrated at the discretion of the team managing the microservice, with the intent of reducing the amount of time between when code changes are made to when they are deployed in production. CI frameworks allow for processes to be executed automatically when code is merged into the main branch, including build operations, unit testing, and integration testing operations. Other CI processes may include validating code style and performing schema evolution validation. A ready-to-deploy container or virtual machine is the final output of the CI pipeline.
持续交付是保持代码库可部署的实践。遵守持续交付原则的微服务使用 CI 管道来验证构建是否已准备好部署。然而,部署本身并不是自动化的,并且需要服务所有者进行一些手动干预。
Continuous delivery is the practice of keeping your codebase deployable. Microservices that adhere to continuous delivery principles use a CI pipeline to validate that the build is ready for deployment. The deployment itself is not automated, however, and requires some manual intervention on the service owner’s part.
持续部署是构建的自动化部署。在端到端持续部署中,提交的代码更改通过 CI 管道传播,达到可交付状态,并根据部署配置自动部署到生产环境。这有助于形成紧凑的开发循环和较短的周转时间,因为已提交的更改可以快速进入生产。
Continuous deployment is the automated deployment of the build. In an end-to-end continuous deployment, a committed code change propagates through the CI pipeline, reaches a deliverable state, and is automatically deployed to production according to the deployment configuration. This contributes to a tight development loop with a short turnaround time, as committed changes quickly enter production.
持续部署在实践中很难做到。有状态服务尤其具有挑战性,因为部署可能需要重建状态存储和重新处理事件流,这对依赖服务尤其具有破坏性。
Continuous deployment is difficult to do in practice. Stateful services are particularly challenging, as deployments may require rebuilding state stores and reprocessing event streams, which are especially disruptive to dependent services.
容器管理系统 (CMS) 提供了管理、部署和控制容器化应用程序的资源使用的方法(请参阅“管理容器和虚拟机”)。CI 过程中构建的容器被存放到存储库中,等待来自 CMS 的部署指令。CI 管道和 CMS 之间的集成对于简化部署过程至关重要,并且通常由所有领先的 CMS 提供商提供,如第2 章中所述。
The container management system (CMS) provides the means of managing, deploying, and controlling the resource use of containerized applications (see “Managing Containers and Virtual Machines”). The container built during the CI process is deposited into a repository, where it awaits deployment instructions from the CMS. Integration between your CI pipeline and the CMS is essential to a streamlined deployment process and is usually provided by all of the leading CMS providers, as discussed in Chapter 2.
商品硬件通常用于部署事件驱动的微服务,因为它价格便宜、性能可靠并且支持服务的水平扩展。您可以根据需要在资源池中添加和删除硬件,而从失败的实例中恢复只需将失败的微服务实例重新部署到新硬件即可。尽管您的微服务实现可能有所不同,但许多事件驱动的微服务不需要任何专用硬件即可运行。对于那些这样做的人,您可以将专门的资源分配到它们自己的独立池中,以便可以相应地部署关联的微服务。示例可能包括用于缓存目的的内存密集型计算实例,或用于需要大量处理能力的应用程序的处理器密集型计算实例。
Commodity hardware is typically used for the deployment of event-driven microservices, as it is inexpensive, performs reliably, and enables horizontal scaling of services. You can add and remove hardware to and from the resource pools as required, while recovery from failed instances requires only that you redeploy the failed microservice instances to the new hardware. Though your microservice implementations may vary, many event-driven microservices do not require any specialized hardware to operate. For those that do, you can allocate specialized resources into their own independent pools, such that the associated microservices can be deployed accordingly. Examples might include memory-intensive computing instances for caching purporses, or processor-intensive computing instances for applications demanding significant processing power.
基本的完全停止部署模式是所有其他模式的基础,本节概述了所涉及的步骤(如图 16-1所示)。根据您的域的具体要求,您的管道中可能有其他步骤,但为了节省空间,我将保持这些步骤的精简。使用您自己的判断和领域知识插入特定于您的用例的任何步骤。
The basic full-stop deployment pattern is the basis of all other patterns, and this section outlines the steps involved (illustrated in Figure 16-1). You may have additional steps in your pipeline depending on the specific requirements of your domain, but I am keeping these steps lean for space purposes. Use your own judgment and domain knowledge to insert any steps specific to your use cases.
提交代码。 将最新代码合并到主分支中,启动 CI 管道。具体细节取决于您的存储库和 CI 管道,但您通常使用提交挂钩来执行此操作,该挂钩可以在代码提交到存储库时执行任意逻辑。
Commit code. Merge the latest code into the master branch, kicking off the CI pipeline. The specifics depend upon your repository and CI pipeline, but you typically do this using commit hooks that can execute arbitrary logic when code is committed to the repository.
执行自动化单元和集成测试。 此步骤是 CI 管道的一部分,用于验证提交的代码是否通过了合并所需的所有单元和集成测试。集成测试可能需要您启动瞬态环境并用数据填充它们以执行更复杂的测试。这需要将 CI 管道与“本地集成测试”中描述的工具集成,以便每个服务都可以启动自己的集成测试环境。
最好为任何类型的自动化集成提供独立的集成测试环境,因为它允许您独立于其他服务运行给定服务的测试。这显着减少了因长期运行和共享的集成测试环境而引起的多租户问题。
Execute automated unit and integration tests. This step is part of the CI pipeline to validate that the committed code passes all the unit and integration tests necessary for merging. Integration tests may require that you spin up transient environments and populate them with data to perform more complex tests. This requires integration of the CI pipeline with the tooling described in “Local Integration Testing” so that each service can bring up its own integration testing environment.
It’s best to have independent integration testing environments for any sort of automated integration, as it allows you to run tests for a given service in isolation from other services. This significantly reduces multitenancy issues that arise from having a long-running and shared integration test environment.
Run predeployment validation tests. This step ensures that your microservice will deploy properly by validating common issues before release. Validations may include:
Validate that input event streams exist, output streams exist (or can be created, if automatic creation is enabled), and your microservice has the proper read/write permissions to access them.
Validate that both the input and output schemas follow schema evolution rules. A simple way to do this is by convention, with your input and output schemas contained within a specific directory structure, along with a map of schemas to event streams. This step of the pipeline can simply ingest the schemas and run the comparisons for you, detecting any incompatibilities.
部署。需要先停止当前部署的微服务,然后才能部署新的微服务。这个过程由两个主要步骤组成:
在部署之前停止实例并执行任何清理工作。停止微服务实例。执行任何必要的状态存储重置和/或消费者组重置,并删除任何内部流。如果在部署失败的情况下重建状态的成本很高,那么您可能希望保留状态、消费者组和内部主题,而将其部署为新服务。这将使您能够在发生故障时快速回滚。
部署。执行实际部署。部署容器化代码并启动所需数量的微服务实例。等待它们启动并发出准备就绪的信号,然后再继续下一步。如果失败,请放弃此步骤并部署代码的先前工作版本。
Deployment. The currently deployed microservice needs to be halted before the new one can be deployed. This process consists of two major steps:
Stop instances and perform any clean-up before deploying. Stop the microservice instances. Perform any necessary state store resets and/or consumer group resets, and delete any internal streams. If rebuilding state in the case deployment failure is expensive, you may instead want to leave your state, consumer group, and internal topics alone, and instead deploy as a new service. This will enable you to roll back quickly in the case of a failure.
Deploy. Perform the actual deployment. Deploy the containerized code and start the required number of microservice instances. Wait for them to boot up and signal that they are ready before moving on to the next step. In the case of a failure, abandon this step and deploy the previous working version of the code.
Run post-deployment validation tests. Validate that the microservice is operating normally, that consumer lag is returning to normal, that there are no logging errors, and that endpoints are working as expected.
考虑对所有相关服务的影响,包括 SLA、停机时间、流处理追赶时间、输出事件负载、新事件流和破坏性架构更改。与相关服务所有者进行沟通,以确保影响是可以接受的。
Consider the impacts to all dependent services, including SLAs, downtime, stream processing catch-up time, output event load, new event streams, and breaking schema changes. Communicate with dependent service owners to ensure that the impacts are acceptable.
滚动更新模式可用于在更新各个微服务实例的同时保持服务运行。其先决条件包括以下内容:
The rolling update pattern can be used to keep a service running while updating the individual microservice instances. Its prerequisites include the following:
没有对任何国有商店进行重大更改
No breaking changes to any state stores
内部微服务拓扑没有重大变化(特别是与使用轻量级框架的实现相关)
No breaking changes to the internal microservice topology (particularly relevant for implementations using lightweight frameworks)
内部事件模式没有重大变化
No breaking changes to internal event schemas
只要满足先决条件,此部署模式就适用于以下场景:
So long as the prerequisites are met, this deployment pattern works well for scenarios such as when:
输入事件中添加了新字段,需要在业务逻辑中体现
New fields have been added to the input events and need to be reflected in the business logic
将消耗新的输入流
New input streams are to be consumed
需要修复错误但不需要重新处理
Bugs need to be fixed but don’t require reprocessing
无意中更改内部微服务拓扑是人们在尝试使用此部署模式时最常见的错误之一。这样做是一项重大更改,需要完全重置应用程序而不是滚动更新。
Inadvertently altering the internal microservice topology is one of the most common mistakes people make when trying to use this deployment pattern. Doing so is a breaking change and will require a full application reset instead of a rolling update.
在滚动更新期间,仅更改“基本完全停止部署模式”的步骤 4。一次仅停止一个实例,而不是同时停止每个实例。然后,停止的实例将被更新并重新启动,以便在部署过程中新旧实例混合运行。这种滚动更新意味着在短时间内,新旧逻辑将同时运行。
During a rolling update, only step 4 of “The Basic Full-Stop Deployment Pattern” is changed. Instead of stopping each instance at the same time, only one instance at a time is stopped. The stopped instance is then updated and started back up, such that a mixture of new and old instances are running during the deployment process. This rolling update means that for a short period of time, both old and new logic will be operating simultaneously.
智能实施将运行测试来检查版本的兼容性,以通知您滚动更新是否有效。手动执行此操作很容易出错,应该避免。
Smart implementations will run a test that checks compatibility of a release to notify you if the rolling update is valid. Doing this manually is quite error-prone and should be avoided.
这种模式的主要好处是可以更新服务,同时近乎实时的处理继续不间断,从而消除停机时间。这种模式的主要缺点是它的先决条件,这限制了它在特定场景的使用。
The main benefit of this pattern is that services can be updated while near-real-time processing continues uninterrupted, eliminating downtime. The main drawback of this pattern is its prerequisites, which limits its usage to specific scenarios.
破坏性架构更改有时是不可避免的,如“破坏性架构更改”中所述。涉及破坏性架构更改的部署必须考虑许多依赖性,包括消费者和生产者的责任、迁移工作的协调以及重新处理停机时间。
A breaking schema change is sometimes inevitable, as covered in “Breaking Schema Changes”. Deployments involving breaking schema changes must take into account a number of dependencies, including both consumer and producer responsibilities, coordination of migration efforts, and reprocessing downtime.
部署重大架构更改是一个相当简单的技术过程。困难的部分是重新协商模式定义、与利益相关者进行沟通以及协调部署和迁移计划。每个步骤都需要各方之间进行清晰的沟通以及明确的行动时间表。
Deploying a breaking schema change is a fairly straightforward technical process. The difficult part is renegotiating the schema definition, communicating that with stakeholders, and coordinating deployment and migration plans. Each of these steps requires clear communication between parties and well-defined timelines for action.
破坏性架构更改的影响因事件类型而异。破坏性实体模式更改比非实体事件更复杂,因为实体需要消费者物化的一致定义。根据定义,实体也是持久的数据单元,每当消费者从其源重新实现实体流时,实体就会被重新使用。实体流必须在新模式下重新创建,并结合新的业务逻辑和模式 定义。
The impacts of a breaking schema change vary depending on the type of event. Breaking entity schema changes are more complex than those of nonentity events, as entities require a consistent definition for consumer materialization. Entities are also, by definition, durable units of data that will be reconsumed whenever a consumer rematerializes the entity stream from its source. Entity streams must be re-created under the new schema, incorporating both the new business logic and schema definitions.
为新流重新创建实体将需要为生产者重新处理必要的源数据,无论是批处理源还是其自己的输入事件流。该逻辑可以构建到同一个生产者中,也可以与其一起构建和部署新的生产者。前一个选项将所有逻辑封装在其自己的服务中,而后一个选项允许原始生产者不间断地继续其操作,从而减少对下游消费者的影响。这些选项如图 16-2所示。
Re-creating the entities for the new stream will require reprocessing the necessary source data for the producer, whether a batch source or its own input event streams. This logic can be built into the same producer, or a new producer can be built and deployed alongside it. The former option keeps all logic encapsulated within its own service, whereas the latter option allows the original producer to continue its operations uninterrupted, reducing impact to downstream consumers. These options are illustrated in Figure 16-2.
破坏性的模式变化通常反映了实体或事件领域的根本性转变。这些通常不会经常发生,但是当它们发生时,变化通常足够大,以至于消费者必须进行更新以反映域的业务含义的变化。
Breaking schema changes often reflect a fundamental shift in the domain of an entity or event. These usually don’t happen too often, but when they do, the change is usually significant enough that consumers must be updated to reflect the shifted business meaning of the domain.
另一方面,非实体事件的重大更改可能不需要重新处理。这主要是因为许多事件流应用程序不会像服务可能需要重新实现其实体流那样定期重新处理事件流。消费者通常可以简单地将新事件定义添加为新事件流,并修改其业务逻辑以处理旧事件和新事件。一旦旧事件从事件流中过期,旧事件流就可以简单地从业务逻辑中删除。
Breaking changes for nonentity events, on the other hand, may not require reprocessing. This is predominantly because many event streaming applications don’t regularly reprocess event streams in the same way that a service may need to rematerialize its entity streams. Consumers can often simply add the new event definition as a new event stream and modify their business logic to handle both old and new events. Once the old events expire out of the event stream, the old event stream can simply be dropped from the business logic.
迁移破坏性架构更改有两个主要选项:
There are two main options for migrating a breaking schema change:
通过两个事件流进行最终迁移,一个使用旧架构,另一个使用新架构
Eventual migration via two event streams, one with the old schema and one with the new schema
同步迁移到单个新流,并删除旧流
Synchronized migration to a single new stream, with the old one removed
通过两个事件流的最终迁移要求生产者以旧格式和新格式将事件写入各自的流。旧流被标记为已弃用,它的使用者将在自己的时间迁移到新流。一旦所有消费者都迁移了,旧的流就可以被删除或卸载到长期数据存储中。
Eventual migration via two event streams requires that the producer write events with both the old and new format, each to its respective stream. The old stream is marked as deprecated, and the consumers of it will, in their own time, migrate to the new stream instead. Once all of the consumers have migrated, the old stream can be removed or offloaded into long-term data storage.
该策略做出了一些假设:
This strategy makes a couple of assumptions:
事件可以生成到旧流和新流。制作者必须拥有必要的数据来创建新旧格式的事件。生成的事件的领域将发生重大变化,足以需要进行重大更改,同时保持足够的相似性,使旧事件格式与新事件格式的 1:1 映射仍然有意义。对于所有破坏性架构更改来说,情况可能并非如此。
Events can be produced to both the old and new streams. The producer must have the necessary data available to create events of both the old and new format. The domain of the produced event will have changed significantly enough to require a breaking change, while remaining similar enough that a 1:1 mapping of old to new event format still makes sense. This may not be the case with all breaking schema changes.
最终迁移不会导致下游不一致。下游服务将继续使用两种不同的定义,但不会产生间接影响,或者这些影响将受到限制。同样,模式中的重大变化表明域已经发生了足够大的变化,以至于组织需要重新定义。这种情况很少见:重大变更对业务来说是必要的,但对使用事件的消费者来说基本上无关紧要。
Eventual migration will not cause downstream inconsistencies. Downstream services will continue consuming two different definitions, but there will not be consequential effects, or those effects will be limited. Again, the breaking change in the schema suggests the domain has been altered enough that the redefinition was necessary to the organization. It is seldom the case that the breaking change is necessary for the business but largely inconsequential to the consumers that use the events.
最终迁移的主要风险之一是迁移永远不会完成,相似但不同的数据流仍然无限期地使用。此外,迁移期间创建的新服务可能会无意中将自己注册为旧流而不是新流上的消费者。使用元数据标记(请参阅“事件流元数据标记”)将流标记为已弃用并保持较小的迁移窗口。
One of the main risks of eventual migration is that the migration is never finished, and similar-yet-different data streams remain in use indefinitely. Additionally, new services created during the migration may inadvertently register themselves as consumers on the old stream instead of the new one. Use metadata tagging (see “Event Stream Metadata Tagging”) to mark streams as deprecated and keep migration windows small.
另一种选择是更新生产者以严格使用新格式创建事件并停止向旧流提供更新。从技术上来说,这是一个比维护两个事件流更简单的选项,但它需要数据的生产者和消费者之间进行更密集的通信。消费者必须更新他们的定义以适应生产者引入的重大变化。
Another option is to update the producer to create events strictly with the new format and to cease providing updates to the old stream. This is a simpler option—technologically speaking—than maintaining two event streams, but it requires more intensive communication between the producer and consumers of the data. Consumers must update their definitions to accommodate breaking changes introduced by the producer.
该策略还做出了一些假设:
This strategy also makes a few assumptions:
事件定义的更改非常重要,旧格式不再可用。实体或事件的领域发生了很大变化,以至于新旧格式无法同时维护。
The event definition change is significant enough that the old format is no longer usable. The domain of the entity or event has changed so much that the old and new format cannot be maintained concurrently.
迁移必须同步进行,以消除下游的不一致。领域变化如此之大,以至于需要更新服务以确保满足业务需求。否则下游服务可能会出现严重不一致。例如,考虑一个实体,其中创建事件的选择标准已更改。
Migration must happen synchronously to eliminate downstream inconsistencies. The domain has changed so significantly that services need to update to ensure that business requirements can be met. Downstream services could have major inconsistencies otherwise. For example, consider an entity where the selection criteria for creation of the event has changed.
此部署计划的最大风险是消费者可能会在迁移到新事件流时失败,但无法像使用最终迁移策略一样优雅地回退到旧数据源。集成测试(最好使用以编程方式生成的环境和源数据)可以通过提供完全执行迁移过程的环境来降低这种风险。您可以在测试环境中一起创建并注册生产者和消费者,以在生产中执行迁移之前验证迁移。
The biggest risk of this deployment plan is that consumers may fail in their migration to the new event stream, but be unable to gracefully fall back to the old source of data as they would using the eventual migration strategy. Integration testing (preferably using programmatically generated environments and source data) can reduce this risk by providing an environment in which to completely exercise the migration process. You can create and register the producer and the consumers together in the test environment to validate the migration prior to performing it in production.
同步迁移在实践中往往很少见,因为它们需要重大的破坏性更改,甚至需要破坏事件之前的域模型。核心业务实体通常具有非常稳定的领域模型,但当发生重大重大变更时,同步迁移可能是不可避免的。
Synchronized migrations tend to be rare in practice, as they require significant breaking changes or even the destruction of the event’s previous domain model. Core business entities usually have very stable domain models, but when major breaking changes occur, a synchronous migration may be unavoidable.
蓝绿部署的主要目标是在部署新功能时提供零停机时间。此模式主要用于同步请求-响应微服务部署,因为它允许同步请求在服务更新时继续。该模式的一个示例如图 16-3所示。
The main goal of blue-green deployment is to provide zero downtime while deploying new functionality. This pattern is predominantly used in synchronous request-response microservice deployments, as it allows for synchronous requests to continue while the service is updated. An example of this pattern is shown in Figure 16-3.
在此示例中,新微服务(蓝色)的完整副本与旧微服务(绿色)并行启动。蓝色服务拥有完全隔离的外部数据存储实例、自己的事件流使用者组以及用于远程访问的自己的 IP 地址。它消耗输入事件,直到监控表明它已充分捕获绿色服务,此时来自绿色实例的流量可以开始重新路由。
In this example, a full copy of the new microservice (blue) is brought up in parallel with the old microservice (green). The blue service has a fully isolated instance of its external data store, its own event stream consumer group, and its own IP addresses for remote access. It consumes input events until monitoring indicates that it is sufficiently caught up to the green service, at which point traffic from the green instances can begin to be rerouted.
流量的切换是由业务前面的路由器来完成的。可以将少量流量转移到新的蓝色实例,以便实时验证部署。如果部署过程没有检测到故障或异常,则可以转移越来越多的流量,直到绿色端不再接收任何流量。
The switchover of traffic is performed by the router in front of the services. A small amount of traffic can be diverted to the new blue instance, such that the deployment can be validated in real time. If the deployment process detects no failures or abnormalities, more and more traffic can be diverted until the green side is no longer receiving any traffic.
此时,根据应用程序的敏感性以及提供快速回退的需要,可以立即关闭绿色实例或使其处于空闲状态,直到经过足够的时间而没有发生事件。如果在此冷却期内发生错误,路由器可以快速将流量重新路由回绿色实例。
At this point, depending on the sensitivity of your application and the need to provide a quick fallback, the green instances can be turned off immediately or left to idle until sufficient time without incident has passed. In the case that an error occurs during this cooldown period, the router can quickly reroute the traffic back to the green instances.
监控和警报(包括资源使用指标、消费者组滞后、自动缩放触发器和系统警报)需要集成为颜色切换过程的一部分。
Monitoring and alerting—including resource usage metrics, consumer group lag, autoscaling triggers, and system alerts—need to be integrated as part of the color switching process.
蓝绿部署非常适合从事件流中使用的微服务。当仅由于请求-响应活动而生成事件时,例如当请求直接转换为事件时,它们也可以很好地工作(请参阅“在事件驱动的工作流中处理请求”)。
Blue-green deployments work well for microservices that consume from event streams. They can also work well when events are produced only due to request-response activity, such as when the request is converted directly into an event (see “Handling Requests Within an Event-Driven Workflow”).
当微服务响应输入事件流向输出流生成事件时,蓝绿部署不起作用。在实体流的情况下,两个微服务将覆盖彼此的结果,在事件流的情况下,这两个微服务将创建重复的事件。请改用滚动更新模式或基本的完全停止部署模式。
Blue-green deployments do not work when the microservice produces events to an output stream in reaction to an input event stream. The two microservices will overwrite each other’s results in the case of entity streams or will create duplicated events in the case of event streams. Use either the rolling update pattern or the basic full-stop deployment pattern instead.
简化微服务的部署需要您的组织缴纳微服务税并投资必要的部署系统。由于可能需要管理大量微服务,因此最好将部署职责委托给拥有微服务的团队。这些团队将需要支持工具来控制和管理他们的部署。
Streamlining the deployment of microservices requires your organization to pay the microservice tax and invest in the necessary deployment systems. Due to the large number of microservices that may need to be managed, it is best to delegate deployment responsibilities to the teams that own the microservices. These teams will need supportive tooling to control and manage their deployments.
持续集成管道是部署过程的重要组成部分。它们提供了用于设置和执行测试、验证构建并确保容器化服务准备好部署到生产的框架。容器管理系统提供了管理容器到计算集群的部署、分配资源和提供可扩展性的方法。
Continuous integration pipelines are an essential part of the deployment process. They provide the framework for setting up and executing tests, validating builds, and ensuring that the containerized services are ready to deploy to production. The container management system provides the means for managing the deployment of the containers into the compute clusters, allocating resources, and providing scalability.
部署服务的方法有很多种,最简单的就是完全停止微服务并重新部署最新的代码。然而,这可能会导致大量停机时间,并且可能不适合,具体取决于 SLA。还有其他几种部署模式,每种模式都有自己的优点和缺点。本章讨论的模式绝不是一个全面的列表,但应该为您确定自己的服务需求提供一个良好的起点。
There are a number of ways to deploy services, with the simplest being to stop the microservice fully and redeploy the newest code. This can incur significant downtime, however, and may not be suitable depending on the SLAs. There are several other deployment patterns, each with its own benefits and drawbacks. The patterns discussed in this chapter are by no means a comprehensive list, but should give you a good starting point for determining the needs of your own services.
事件驱动的微服务架构提供了一种强大、灵活且定义明确的方法来解决业务问题。以下是对本书所涵盖内容的快速回顾,以及一些最后的话。
Event-driven microservice architectures provide a powerful, flexible, and well-defined approach to solving business problems. Here is a quick recap of the things that have been covered in this book, as well as some final words.
数据通信结构允许对整个组织内的重要业务事件进行通用访问。事件代理非常好地满足了这一需求,因为它们允许严格的数据组织,可以近乎实时地传播更新,并且可以在大数据规模下运行。数据通信与转换和利用数据的业务逻辑严格分离,将这些需求卸载到单独的有界上下文中。这种关注点分离使得事件代理能够在很大程度上不了解业务逻辑需求(除了支持读取和写入之外),从而使其能够严格专注于存储、保存事件数据并将其分发给消费者。
The data communication structure allows for universal access of important business events across the organization. Event brokers fulfill this need extremely well, as they permit the strict organization of data, can propagate updates in near–real time, and can operate at big-data scale. The communication of data remains strictly decoupled from the business logic that transforms and utilizes it, offloading these requirements into individual bounded contexts. This separation of concerns allows the event broker to remain largely agnostic of the business logic requirements (aside from supporting reads and writes), enabling it focus strictly on storing, preserving, and distributing the event data to consumers.
成熟的数据通信层将数据的所有权和生产与其访问和消费解耦。应用程序不再需要执行双重职责:为内部业务逻辑提供服务,同时还为其他服务提供同步机制和外部直接访问。
A mature data communication layer decouples the ownership and production of data from the access and consumption of it. Applications no longer need to perform double duty by serving internal business logic while also providing synchronization mechanisms and outside direct access for other services.
任何服务都可以利用事件代理的持久性和弹性来使其数据高度可用,包括那些使用事件代理来存储其内部状态更改日志的服务。服务实例失败不再意味着数据无法访问,而只是意味着新数据将被延迟,直到生产者重新上线。同时,在任何生产者中断期间,消费者可以自由地从事件代理进行消费,从而解耦服务之间的故障模式。
Any service can leverage the durability and resiliency of the event broker to makes its data highly available, including those that use the event broker to store changelogs of its internal state. A failed service instance no longer means that data is inaccessible, but simply that new data will be delayed until the producer is back online. Meanwhile, consumers are free to consume from the event broker during any producer outages, decoupling the failure modes between the services.
企业在特定的域中运营,该域可以分解为子域。业务问题的解决方案由有界上下文提供,它识别与子域相关的边界,包括输入、输出、事件、需求、流程和数据模型。
Businesses operate in a specific domain, which can be broken down into subdomains. Solutions to business problems are informed by bounded contexts, which identify the boundaries—including the inputs, outputs, events, requirements, processes, and data models—relevant to the subdomain.
可以构建微服务实现来与这些有界上下文保持一致。由此产生的服务和工作流程与问题和业务需求相一致。通用数据通信层通过确保微服务实现足够灵活以适应业务领域和后续有界上下文中的变化来帮助促进这些一致性。
Microservice implementations can be built to align with these bounded contexts. The resultant services and workflows thus align with the problems and business requirements. The universal data communication layer helps facilitate these alignments by ensuring that the microservice implementations are flexible enough to adapt to changes within the business domain and subsequent bounded contexts.
事件驱动的微服务需要对允许其大规模运行的系统和工具进行投资,称为微服务税。事件代理是系统的核心,因为它提供服务之间的基本通信,并使每个服务无需管理自己的数据通信解决方案。
Event-driven microservices require an investment in the systems and tools that permit its operation at scale, known as the microservice tax. The event broker is at the heart of the system, as it provides the fundamental communication between services and absolves each service from managing its own data communication solution.
微服务架构放大了围绕创建、管理和部署应用程序的所有问题,并从这些流程的标准化和简化中受益。随着微服务数量的增长,这一点变得更加重要,因为虽然每个新的微服务都会产生开销,但非标准化的微服务会比遵循协议的微服务产生更多的开销。
Microservice architectures amplify all of the issues surrounding creating, managing, and deploying applications, and benefit from a standardization and streamlining of these processes. This becomes more critical as the number of microservices grow, because while each new microservice incurs an overhead, the nonstandardized ones can incur significantly more than those following a protocol.
构成微服务税的基本服务包括:
The essential services that make up the microservice tax include:
事件经纪人
The event broker
模式注册和数据探索服务
Schema registry and data exploration service
集装箱管理系统
Container management system
持续集成、交付和部署服务
Continuous integration, delivery, and deployment service
监控和记录服务
Monitoring and logging service
支付微服务税不太可能是一个全有或全无的过程。组织通常会从事件代理服务或容器管理系统开始,然后根据需要添加其他部分。幸运的是,许多计算服务提供商(例如 Google、Microsoft 和 Amazon)已经创建了可显着降低开销的服务。您必须权衡您的选择,并选择是将这些操作外包给服务提供商还是在内部构建自己的系统。
Paying the microservice tax is unlikely to be an all-or-nothing process. An organization will typically start with either an event broker service or a container management system, and work toward adding other pieces as needed. Fortunately, a number of computing service providers, such as Google, Microsoft, and Amazon, have created services that significantly reduce the overhead. You must weigh your options and choose between outsourcing these operations to a service provider or building your own systems in-house.
图式在传达事件含义方面发挥着关键作用。强类型事件迫使生产者和消费者面对数据的现实。生产者必须确保他们根据模式生成事件,而消费者必须确保他们处理所消费事件的类型、范围和定义。严格定义的模式显着减少了消费者误解事件的机会,并为未来的变化提供了契约。
Schemas play a pivotal role in communicating the meaning of events. Strongly typed events force both producers and consumers to contend with the realities of the data. Producers must ensure that they produce events according to the schema, while consumers must ensure that they handle the types, ranges, and definitions of the consumed events. Strongly defined schemas significantly reduce the chance of consumers misinterpreting events and provide a contract for future changes.
模式演化为事件和实体提供了一种机制,可以根据新的业务需求进行更改。它使生产者能够使用新的和更改的字段生成数据,同时还允许不关心更改的消费者继续使用旧的模式。这显着降低了不必要变更的频率和风险。同时,其余的消费者可以独立升级其代码,以使用最新的模式格式来消费和处理事件,从而使他们能够访问最新的数据字段。
Schema evolution provides a mechanism for events and entities to change in response to new business requirements. It enables the producer to generate data with new and altered fields, while also allowing consumers that don’t care about the change to continue using older schemas. This significantly reduces the frequency and risk of nonessential change. Meanwhile, the remaining consumers can independently upgrade their code to consume and process events using the latest schema format, enabling them to access the latest data fields.
最后,模式还允许有用的功能,例如代码生成和数据发现。代码生成器可以创建与生产和使用应用程序相关的类/结构。这些强定义的类/结构可以帮助在编译时或本地测试期间检测生产和消费中的格式错误,确保事件可以无缝地生产和消费。这使得开发人员能够严格关注处理和转换这些事件的业务逻辑,而不是关注管道。同时,模式注册表提供了一种可搜索的方法来确定哪些数据属于哪个事件流,从而可以更轻松地发现流的内容。
Finally, schemas also allow for useful features such as code generation and data discovery. Code generators can create classes/structures relevant to producing and consuming applications. These strongly defined classes/structures can help detect formatting errors in production and consumption either at compile time or during local testing, ensuring that events can be seamlessly produced and consumed. This allows developers to focus strictly on the business logic of handling and transforming these events, and far less on the plumbing. Meanwhile, a schema registry provides a searchable means of figuring out which data is attributed to which event stream, enabling easier discovery of the stream’s contents.
一旦数据通信层可用,就可以将关键业务数据放入其中。从组织中的各种服务和数据存储中释放数据可能是一个漫长的过程,并且需要一些时间才能将所有必要的数据获取到事件代理中。这是系统相互解耦以及转向事件驱动架构的重要一步。数据解放将数据的生产和所有权与下游消费者的访问分离。
Once the data communication layer is available, it’s time to get the business-critical data into it. Liberating data from the various services and data stores in your organization can be a lengthy process, and it will take some time to get all necessary data into the event broker. This is an essential step in decoupling systems from one another and for moving toward an event-driven architecture. Data liberation decouples the production and ownership of data from the accessing of it by downstream consumers.
首先释放最常用且对组织下一个主要目标最关键的数据。从各种服务和数据存储中提取信息的方法有多种,每种方法都有优点和缺点。重要的是要权衡对现有服务的影响与陈旧数据、缺乏模式以及释放的事件流中内部数据模型暴露的风险。
Start by liberating the data that is most commonly used and most critical to your organization’s next major goals. There are various ways to extract information from the various services and data stores, and each method has benefits and drawbacks. It’s important to weigh the impact on the existing service against the risks of stale data, lack of schemas, and exposure of internal data models in the liberated event streams.
拥有事件流形式的现成可用业务数据允许通过组合来构建服务。新服务只需要通过事件代理订阅感兴趣的事件流,而不是直接连接到本来提供数据的每个服务。
Having readily available business data in the form of event streams allows for services to be built by composition. A new service needs only to subscribe to the event streams of interest via the event broker, rather than directly connecting to each service that would otherwise provide the data.
微服务作为有界上下文的实现,专注于解决有界上下文的业务问题,并进行相应的调整。业务需求变化是微服务更新的主要驱动力,所有其他不相关的微服务保持不变。
As the implementation of the bounded context, microservices are focused on solving the business problems of the bounded context, and aligned accordingly. Business requirement changes are the primary driver of updates to a microservice, with all other unrelated microservices remaining unchanged.
避免基于技术边界实施微服务。这些通常是作为服务多个业务工作流程的短期优化而创建的,但这样做时,它们将自身耦合到每个工作流程。技术微服务对业务变化变得敏感,并将原本不相关的工作流程耦合在一起。技术微服务的失败或无意的更改可能会导致多个业务工作流程瘫痪。尽可能避免技术一致性,而专注于实现业务的有界上下文。
Avoid implementing microservices based on technical boundaries. These are generally created as a short-term optimization for serving multiple business workflows, but in doing so, they couple themselves to each workflow. The technical microservice becomes sensitive to business changes and couples otherwise unrelated workflows together. A failure or inadvertent change in the technical microservice can bring down multiple business workflows. Steer clear of technical alignment whenever possible, and focus instead on fulfilling the business’s bounded context.
最后,并非所有微服务都需要是“微”的。对于一个组织来说,使用多个较大的服务是合理的,特别是在它尚未部分或全部缴纳微服务税的情况下。这是组织架构的正常演变。如果您的企业正在使用大量大型服务,遵循以下原则将有助于创建与现有大型服务分离的新的细粒度服务:
Lastly, not all microservices need be “micro.” It is reasonable for an organization to instead use several larger services, particularly if it has not paid the microservice tax, in part or in full. This is a normal evolution of an organization’s architecture. If your business is using numerous large services, following these principles will help enable the creation of new, fine-grained services decoupled from the existing large services:
将重要的业务实体和事件放入事件代理中。
Put important business entities and events into the event broker.
使用事件代理作为单一事实来源。
Use the event broker as the single source of truth.
避免在服务之间使用直接调用。
Avoid using direct calls between services.
对于构建事件驱动的微服务,有多种可用选项,各有利弊。目前,轻量级框架往往具有最好的开箱即用功能。流可以具体化为表并无限期地维护。可以对许多流和表执行连接(包括外键上的连接)。热副本、持久存储和变更日志提供了弹性和可扩展性。
There’s a wide range of options available, all with pros and cons, for building event-driven microservices. Currently, lightweight frameworks tend to have the greatest out-of-the-box functionality. Streams can be materialized to tables and maintained indefinitely. Joins, including those on foreign keys, can be performed for numerous streams and tables. Hot replicas, durable storage, and changelogs provide resiliency and scalability.
重量级框架提供与轻量级框架类似的功能,但在处理独立性方面存在不足,因为它们需要单独的专用资源集群。新的集群管理选项变得越来越流行,例如与领先的容器管理系统之一 Kubernetes 直接集成。重量级框架通常已经在中型和大型组织中使用,通常由执行大数据分析的个人使用。
Heavyweight frameworks provide similar functionality to lightweight frameworks, but fall short in terms of processing independence because they require a separate dedicated cluster of resources. New cluster management options are becoming more popular, such as direct integration with Kubernetes, one of the leading container management systems. Heavyweight frameworks are often already in use in mid-sized and larger organizations, typically by the individuals performing big-data analysis.
基本生产者/消费者 (BPC) 和功能即服务 (FaaS) 解决方案为多种语言和运行时提供了灵活的选项。这两种选择都仅限于基本消费和生产模式。确保确定性行为很困难,因为这两种选项都没有内置事件调度,因此复杂的操作需要对您自己的自定义库进行大量投资才能提供该功能,或者将使用限制为简单的用例模式。
Basic producer/consumer (BPC) and Function-as-a-Service (FaaS) solutions provide flexible options for many languages and runtimes. Both options are limited to basic consumption and production patterns. Ensuring deterministic behavior is difficult, as neither option comes with built-in event scheduling, so complex operations require either a significant investment in your own custom libraries to provide that functionality, or limiting the usage to simple use-case patterns.
事件驱动的微服务非常适合完全集成和单元测试。作为事件驱动微服务的主要输入形式,事件可以轻松组合以涵盖可能出现的任何情况。事件模式限制了需要测试的值的范围,并提供了组成输入测试流所需的结构。
Event-driven microservices lend themselves very well to full integration and unit testing. As the predominant form of inputs to an event-driven microservice, events can be easily composed to cover whatever cases may arise. Event schemas constrain the range of values that need to be tested and provide the necessary structure for composing input test streams.
本地测试可以包括单元测试和集成测试,后者依赖于事件代理、模式注册表以及被测服务所需的任何其他依赖项的动态创建。例如,可以创建事件代理以在与测试相同的可执行文件中运行(就像许多基于 JVM 的解决方案一样),或者在其自己的容器中与被测应用程序一起运行。通过完全控制事件代理,您可以模拟负载、计时条件、中断、故障或任何其他代理应用程序交互。
Local testing can include both unit and integration testing, with the latter relying on the dynamic creation of an event broker, schema registry, and any other dependencies required by the service under test. For example, an event broker can be created to run within the same executable as the test, as is the case with numerous JVM-based solutions, or to run within its own container alongside the application under test. With full control of the event broker, you can simulate load, timing conditions, outages, failures, or any other broker-application interactions.
您可以通过动态创建临时事件代理集群、用生产事件流和事件的副本填充其中(排除信息安全问题)并针对其执行应用程序来进行生产集成测试。这可以在生产部署之前提供烟雾测试,以确保没有遗漏任何内容。您还可以在此环境中执行性能测试,测试单个实例的性能以及应用程序水平扩展的能力。测试完成后,可以简单地丢弃该环境。
You can conduct production integration testing by dynamically creating a temporary event broker cluster, populating it with copies of production event streams and events (barring information-security concerns) and executing the application against it. This can provide a smoke test prior to production deployment to ensure that nothing has been overlooked. You can also execute performance tests in this environment, testing both the performance of a single instance and the ability of the application to horizontally scale. The environment can simply be discarded once testing is complete.
大规模部署微服务要求微服务所有者能够快速轻松地部署和回滚其服务。这种自主权允许团队独立移动和行动,并消除单独负责部署的基础设施团队中可能存在的瓶颈。持续集成、交付和部署管道对于提供此功能至关重要,因为它们允许简化但可定制的部署流程,减少手动步骤和干预,并且可以扩展到其他微服务。根据您的选择,容器管理系统可能会提供额外的功能来帮助部署和回滚,从而进一步简化流程。
Deploying microservices at scale requires that microservice owners can quickly and easily deploy and roll back their services. This autonomy allows for teams to move and act independently and to eliminate bottlenecks that would otherwise exist in an infrastructural team responsible solely for deployments. Continuous integration, delivery, and deployment pipelines are essential in providing this functionality, as they allow a streamlined yet customizable deployment process that reduces manual steps and intervention, and can be scaled out to other microservices. Depending on your selection, the container management system may provide additional functionality to help with deployments and rollbacks, further simplifying the process.
部署过程必须考虑服务级别协议 (SLA)、状态重建和输入事件流的重新使用。SLA 不仅仅是停机问题,还必须考虑部署对所有下游消费者的影响以及事件代理服务的运行状况。必须重建其整个状态并传播新输出事件的微服务可能会给事件代理带来相当大的负载,并导致下游使用者突然需要扩展到更多的处理实例。重建服务在短时间内处理数百万或数十亿个事件并不罕见。配额可以减轻影响,但根据下游服务要求,重建服务可能会在不可接受的时间内处于不一致状态。
The deployment process must take into account service-level agreements (SLAs), rebuilding of state, and reconsumption of input event streams. SLAs are not simply matters of downtime, but also must take into account the impact of deployment on all downstream consumers, and the health of the event broker service. A microservice that must rebuild its entire state and propagate new output events may place a considerable load on the event broker, as well as cause downstream consumers to suddenly need to scale up to many more processing instances. It is not uncommon for a rebuilding service to process millions or billions of events in short order. Quotas can mitigate the impact, but depending on downstream service requirements, a rebuilding service may be in an inconsistent state for an unacceptable period of time.
SLA、对下游消费者的影响、对事件代理的影响以及对监控和警报框架的影响之间总是存在权衡。例如,蓝绿部署需要两个消费者组,必须考虑监控和警报,包括基于滞后的自动缩放。虽然您当然可以执行适应此部署模式所需的工作,但另一种选择是简单地更改应用程序的设计。蓝绿部署的替代方案是使用薄的、始终在线的服务层来服务同步请求,而后端事件处理器可以在自己的时间内换出并重新处理。虽然您的服务层可能会在一段时间内提供过时的数据,但它不需要任何工具增强或更复杂的交换操作,并且仍然可能满足依赖服务的 SLA。
There are always tradeoffs between SLAs, impact to downstream consumers, impact to the event broker, and impact to monitoring and alerting frameworks. For instance, a blue-green deployment requires two consumer groups, which must be considered for monitoring and alerting, including lag-based autoscaling. While you can certainly perform the work necessary to accommodate this deployment pattern, another option is to simply change the application’s design. An alternative to blue-green deployments is to use a thin, always-on serving layer to serve synchronous requests, while the backend event processor can be swapped out and reprocess in its own time. While your service layer may serve stale data for a period of time, it doesn’t require any augmentations to tooling or more complex swapping operations and can possibly still meet the SLAs of dependent services.
事件驱动的微服务要求您重新思考数据到底是什么以及服务如何访问和使用数据。任何特定领域的数据量每年都在飞速增长,并且没有停止的迹象。数据变得越来越大、越来越普遍,可以简单地将数据推入一个大型数据存储并用于所有目的的日子已经一去不复返了。强大且定义良好的数据通信层可以减轻服务执行双重职责的负担,并使它们能够专注于仅服务自己的业务功能,而不是其他有界上下文的数据和查询需求。
Event-driven microservices require you to rethink what data really is and how services go about accessing and using it. The amount of data attributed to any specific domain grows in leaps and bounds every year and shows no signs of stopping. Data is becoming larger and more ubiquitous, and gone are the days when it could simply be shoved into one large data store and used for all purposes. A robust and well-defined data communication layer relieves services of performing double duty and allows them to focus instead on serving just their own business functionality, not the data and querying needs of other bounded contexts.
事件驱动的微服务是处理这些大型且多样化的数据集的计算的自然演变。事件流的组合性质使其具有无与伦比的灵活性,并允许各个业务部门专注于使用实现其业务目标所需的任何数据。运行少量服务的组织可以从事件代理提供的数据通信中受益匪浅,这为构建与旧业务需求和实现完全解耦的新服务铺平了道路。
Event-driven microservices are a natural evolution of computing to handle these large and diverse data sets. The compositional nature of event streams lends itself to unparalleled flexibility and allows individual business units to focus on using whatever data is necessary to accomplish their business goals. Organizations running a handful of services can benefit greatly from the data communications provided by the event broker, which paves the way for building new services, fully decoupled from old business requirements and implementations.
无论事件驱动的微服务本身的未来如何,有一点是明确的:数据通信层将组织数据的力量扩展到需要它的任何服务或团队,消除访问边界,并减少与生产和分发相关的不必要的复杂性的重要商业 信息。
Regardless of the future of event-driven microservices themselves, this much is clear: The data communication layer extends the power of an organization’s data to any service or team that requires it, eliminates access boundaries, and reduces unnecessary complexity related to the production and distribution of important business information.
《构建事件驱动的微服务》封面上的动物是黄颊山雀 ( Machlolophus spilonotus )。这种鸟可以在东南亚的阔叶林和混山林以及人造公园和花园中找到。
The animal on the cover of Building Event-Driven Microservices is a yellow-cheeked tit (Machlolophus spilonotus). This bird can be found in the broadleaf and mixed-hill forests, as well as in the human-made parks and gardens, of southeast Asia.
黄颊山雀引人注目的亮黄色脸部和颈背与黑色的羽冠、喉咙和胸部形成鲜明对比,使其很容易被识别。封面上描绘的雄性有灰色的身体和黑色的翅膀,翅膀上布满了白色的斑点和条纹;雌性有橄榄色的身体和淡黄色的翼条。
The striking bright yellow face and nape of the yellow-cheeked tit in contrast with its black crest, throat, and breast make it easily identifiable. The male, depicted on the cover, has a gray body and black wings peppered with white spots and bars; the female has an olive-colored body and pale yellow wing-bars.
黄颊山雀以小型无脊椎动物、蜘蛛以及一些水果和浆果为食,在森林的低层和中层觅食。与山雀、山雀和山雀科的其他鸟类一样,黄颊山雀通过快速拍动翅膀进行短距离、起伏的飞行。
Yellow-cheeked tits dine on small invertebrates, spiders, and some fruits and berries, foraging in the low- and mid-levels of the forest. Like other birds in the chickadee, tit, and titmice family, the yellow-cheeked tit travels via short, undulating flights with rapidy fluttering wings.
虽然黄颊山雀的对话状态被列为“最不关心”,但 O'Reilly 封面上的许多动物都处于濒危状态;所有这些对世界都很重要。
While the yellow-cheeked tit’s conversation status is listed as of Least Concern, many of the animals on O’Reilly covers are endangered; all of them are important to the world.
封面插图由凯伦·蒙哥马利 (Karen Montgomery) 创作,以动画自然绘画博物馆的黑白版画为基础。封面字体为 Gilroy Semibold 和 Guardian Sans。文字字体为Adobe Minion Pro;标题字体为 Adobe Myriad Condensed;代码字体是Dalton Maag的Ubuntu Mono。
The cover illustration is by Karen Montgomery, based on a black and white engraving from Pictorial Museum of Animated Nature. The cover fonts are Gilroy Semibold and Guardian Sans. The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono.